My company has a blazor server application running on .NET 6.0.Currently I am transitioning the application from a Windows Server 2019 running on IIS to an Azure Web App Service.
Everything runs fine. The app in the new environment is basically as performant as the old Windows Server environment, besides one very specific issue:
It takes way more time to load static assets from wwwroot directory than the Windows Server environment.
Just for a comparison, this is the Network tab on google Chrome showing load times for our Login page.
This is on Windows Server hosting:
Windows Server Login page load time
This is on Azure Web App hosting
as you can see, depending on the file, azure takes a good amount of time more than our windows server.
If we go more in depth, we see the all that extra time is spent waiting for the server to respond to the request (TTFB).
I've been all around the internet for some problem alike. I've also been bombarding azure copilot and other LLM tools to see if one of them give me a light of hope, but none seem to be able to help me in this endeavor, only giving me very generic trobleshooting responses.
I have tried to tinker with the configuration of the web app, and it is looking like this:
Stack Settings
Stack - .NET.NET Version - .NET 6 (LTS)
Platform Settings
Platform - 64 bits
FTP Basic Auth Publishin - On
FTP State - FTPS Only
Inbound IP mode - IPv4
HTTP Version - 2.0
HTTP 2.0 Proxy - Off
Web Sockets - On
Always On - On
Session Affinity - On
HTTPS Only - On
Minimum Inbound TLS Version - 1.2
Minimum Inbound TLS Cipher Suite - TLS_RSA_WITH_AES_128_CBC_SHA
End-to-end TLS encryption - On
Debugging
Remote Debugging - Off
Incoming client certificates
Client certificate mode - Ignore
I have tried to ping and traceroute over my application IP and there is no issues here, so I don't believe it has something to do with networking.
The web app and my computer are both in the same region so I don't think I need any CDN or Azure Front Door set up.
I do have application insights, and it shows it takes a long time, but all other metric seems fine and the information it gives me is very inconclusive.
Looking deeper into one of these requests, I can see that is basically the wait time for the resource:
this is the profilling of a GET request to this specific asset: /app-assets/css/bootstrap-extended.css
as you can see, for this particular request, there was a 15 seconds (!!!) wait time.
I don't have a lot of users on this environment. It's only me and another developer for now, so the resources are not used to their maximum.
We use a P1mv3 app service (2 vCores and 16GB of memory), so I don't think we need to scale up for this.
Is there something I am missing here? Why static resources would take so long to download?
I tried to take compression out and it didn't help. I added Local cache and it didn't help.
Does any one have any clue of what else could I try?
Edit:
This is our Program.cs file for reference:
using Microsoft.AspNetCore.Components.Authorization;using Microsoft.EntityFrameworkCore;using Microsoft.AspNetCore.OData;using Newtonsoft.Json.Converters;using Newtonsoft.Json.Serialization;using Newtonsoft.Json;using Syncfusion.Blazor;using Web.Users;using Web4you.Web.Core;using Web4you.Data;using Microsoft.OData.Edm;using Microsoft.OData.ModelBuilder;using Microsoft.AspNetCore.Mvc;using System.Collections;using System.IO.Abstractions;using CurrieTechnologies.Razor.SweetAlert2;using MassTransit;using Microsoft.AspNetCore.Authorization;using Microsoft.AspNetCore.Identity;using Web4you.Web.Configuration;using Web4you.Web.Areas.Identity;using Microsoft.Extensions.FileProviders;using Web.Backend.Core;using Web.Backend.Core.Helpers;using Microsoft.AspNetCore.ResponseCompression;using Hangfire;using Hangfire.Console;using Hangfire.Mongo;using Hangfire.Mongo.Migration.Strategies;using Hangfire.Mongo.Migration.Strategies.Backup;using Hangfire.SqlServer;using Microsoft.AspNetCore.StaticFiles;using Microsoft.OpenApi.Models;using MongoDB.Driver;using Web.Backend.Core.Jobs;using MudBlazor.Services;using Web.Backend.Requests.Jobs;using Web.Backend.Core.SignalR;using Serilog;var builder = WebApplication.CreateBuilder(args);builder.Configuration.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);builder.Configuration.AddJsonFile($"appsettings.{builder.Environment.EnvironmentName}.json", optional: true, reloadOnChange: true);builder.Configuration.AddJsonFile("appsettings.Local.json", optional: true, reloadOnChange: true);var connectionString = builder.Configuration.GetConnectionString("DefaultConnection");builder.Services.AddDbContext<RepositoryContext>(options => options.UseSqlServer(connectionString));builder.Services.AddDbContextFactory<RepositoryContext>(options => options.UseSqlServer("DefaultConnection"), ServiceLifetime.Scoped);builder.Services.AddDatabaseDeveloperPageExceptionFilter();builder.Services.AddDefaultIdentity<User>(options => options.SignIn.RequireConfirmedAccount = false) .AddRoles<Role>() .AddEntityFrameworkStores<RepositoryContext>();builder.Services.Configure<IdentityOptions>(options =>{ //Regular Identity Options, I am taking this out for sercurity reasons});builder.Services.AddRazorPages();builder.Services.AddServerSideBlazor();builder.Services.AddScoped<AuthenticationStateProvider, RevalidatingIdentityAuthenticationStateProvider<User>>();builder.Services.AddScoped<Config>();builder.Services.AddScoped<IEmailHelper, EmailHelper>();builder.Services.AddScoped<ISmsHelper, SmsHelper>();builder.Services.AddScoped<CaptchaVerificationService>();builder.Services.AddScoped<AuthHelper>();builder.Services.AddScoped<UploadHelper>();builder.Services.AddScoped<StripeHelper>();builder.Services.AddScoped<CommonHelper>();builder.Services.AddScoped<ImportHelper>();builder.Services.AddScoped<QuickbooksHelper>();builder.Services.AddScoped(provider => new Lazy<QuickbooksHelper>(provider.GetRequiredService<QuickbooksHelper>));builder.Services.AddScoped<UserOptions>();builder.Services.AddScoped<UiUpdatesService>();builder.Services.AddScoped<StateContainer>();builder.Services.AddScoped<IAuthorizationHandler, EditAuthorizationHandler>();builder.Services.AddScoped<IAuthorizationHandler, ViewAuthorizationHandler>();builder.Services.AddSingleton<IContentTypeProvider, FileExtensionContentTypeProvider>();builder.Services.AddScoped<HubConnectionManager>();builder.Services.AddSingleton<IStripeWebhookEventWrapper, StripeWebhookEventWrapper>();builder.Services.AddSingleton<IFileSystem, FileSystem>();builder.Services.AddCoreServices();builder.Services.AddHttpContextAccessor();builder.Services.AddHttpClient();builder.Services.AddMudServices();builder.Services.Configure<CaptchaConfig>(builder.Configuration.GetSection(CaptchaConfig.Position));builder.Services.Configure<CryptoConfig>(builder.Configuration.GetSection(CryptoConfig.Position));builder.Services.Configure<PathsConfig>(builder.Configuration.GetSection(PathsConfig.Position));builder.Services.Configure<CommonConfig>(builder.Configuration.GetSection(CommonConfig.Position));builder.Services.Configure<SmtpConfig>(builder.Configuration.GetSection(SmtpConfig.Position));builder.Services.Configure<StripeConfig>(builder.Configuration.GetSection(StripeConfig.Position));builder.Services.Configure<GoogleConfig>(builder.Configuration.GetSection(GoogleConfig.Position));builder.Services.Configure<PlansConfig>(builder.Configuration.GetSection(PlansConfig.Position));builder.Services.Configure<QuickBookConfig>(builder.Configuration.GetSection(QuickBookConfig.Position));builder.Services.Configure<CDyneConfig>(builder.Configuration.GetSection(CDyneConfig.Position));builder.Services.Configure<DemoClientsConfig>(builder.Configuration.GetSection(DemoClientsConfig.Position));builder.Services.Configure<ReCaptchaConfig>(builder.Configuration.GetSection(ReCaptchaConfig.Position));builder.Services.Configure<ProductFruits>(builder.Configuration.GetSection(ProductFruits.Position));builder.Services.Configure<BackgroundJobsCronConfig>(builder.Configuration.GetSection(BackgroundJobsCronConfig.Position));builder.Services.AddServerSideBlazor().AddCircuitOptions(options => { options.DetailedErrors = true; });builder.Services.AddServerSideBlazor().AddHubOptions(o => { o.MaximumReceiveMessageSize = 102400000; });if (builder.Environment.EnvironmentName == Environments.Production){ var mongoUrlBuilder = new MongoUrlBuilder(builder.Configuration.GetConnectionString("HangfireConnection")); var mongoClient = new MongoClient(mongoUrlBuilder.ToMongoUrl()); builder.Services.AddHangfire(configuration => { configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170) .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseMongoStorage(mongoClient, mongoUrlBuilder.DatabaseName, new MongoStorageOptions { MigrationOptions = new MongoMigrationOptions { MigrationStrategy = new MigrateMongoMigrationStrategy(), BackupStrategy = new CollectionMongoBackupStrategy() }, Prefix = "hangfire.mongo", CheckConnection = true, CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.TailNotificationsCollection }); configuration.UseConsole(); });}else{ var hangfireConnectionString = builder.Configuration.GetConnectionString("HangfireConnection"); builder.Services.AddHangfire(configuration => { configuration.SetDataCompatibilityLevel(CompatibilityLevel.Version_170) .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage(hangfireConnectionString, new SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.FromMinutes(5), UseRecommendedIsolationLevel = true, DisableGlobalLocks = true, PrepareSchemaIfNecessary = false }); configuration.UseConsole(); });}builder.Services.AddHangfireServer(options =>{ options.WorkerCount = Environment.ProcessorCount * 5;});builder.Services.AddResponseCompression(opts =>{ opts.MimeTypes = ResponseCompressionDefaults.MimeTypes.Concat( new[] { "application/octet-stream" });});builder.Services.AddControllers() .AddOData(options => { options.AddRouteComponents("odata", GetEdmModel()); options.Select().Filter().Count().OrderBy().Expand(); }) .AddNewtonsoftJson(options => { options.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore; options.SerializerSettings.Converters.Add(new StringEnumConverter()); options.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver(); options.SerializerSettings.DateTimeZoneHandling = DateTimeZoneHandling.Utc; });builder.Services.AddSyncfusionBlazor();var syncfusion = builder.Configuration.GetSection(SyncfusionLicense.Position).Get<SyncfusionLicense>();Syncfusion.Licensing.SyncfusionLicenseProvider.RegisterLicense(syncfusion.Key);builder.Services.AddSweetAlert2();builder.Services.AddConsumers();builder.Services.AddMassTransitHostedService();builder.Services.AddSwaggerGen(c =>{ c.SwaggerDoc("v2", new OpenApiInfo { Title = "ServiceDeck API", Version = "v2" });});builder.Services.AddAuthorization(options =>{ foreach (var m in typeof(ModuleType).GetFields().Select(p => p.Name)) foreach (var p in typeof(PermissionType).GetFields().Select(p => p.Name)) { var name = $"{p}:{m}"; options.AddPolicy(name, policy => policy.RequireAssertion(context => context.User.IsInRole(UserRoles.Admin) || context.User.HasClaim(c => c.Type == name))); }});builder.Services.AddSignalR(o =>{ o.MaximumReceiveMessageSize = null; o.EnableDetailedErrors = true;});builder.Services.AddCors(options =>{ options.AddPolicy("WidgetPolicy", policy => { policy.AllowAnyOrigin() .AllowAnyHeader() .AllowAnyMethod(); });});builder.Services.AddAuthentication().AddGoogle(googleOptions =>{ var googleConfig = builder.Configuration.GetSection(GoogleConfig.Position).Get<GoogleConfig>(); googleOptions.ClientId = googleConfig.OAuthClientId; googleOptions.ClientSecret = googleConfig.OAuthClientSecret;});Log.Logger = new LoggerConfiguration() .ReadFrom.Configuration(builder.Configuration) .CreateLogger();builder.Logging.AddSerilog();var app = builder.Build();app.Map(new PathString(Constants.TemporaryExportUriBase), client =>{ var exportsPath = Path.Combine(Path.GetTempPath(), Constants.TemporaryExportDirectoryName); if (!Directory.Exists(exportsPath)) Directory.CreateDirectory(exportsPath); client.UseStaticFiles(new StaticFileOptions() { FileProvider = new PhysicalFileProvider(exportsPath) });});app.Map(new PathString(Constants.UploadUriBase), client =>{ var pathConfig = builder.Configuration.GetSection(PathsConfig.Position).Get<PathsConfig>(); var imageDir = Path.GetFullPath(pathConfig.ImagesRootPath); if (!Directory.Exists(imageDir)) Directory.CreateDirectory(imageDir); client.UseStaticFiles(new StaticFileOptions() { FileProvider = new PhysicalFileProvider(imageDir) });});app.Map(new PathString(Constants.FileManagerUriBase), client =>{ var pathConfig = builder.Configuration.GetSection(PathsConfig.Position).Get<PathsConfig>(); var fileManagerPath = Path.GetFullPath(pathConfig.FileManagerPath); if (!Directory.Exists(fileManagerPath)) Directory.CreateDirectory(fileManagerPath); client.UseStaticFiles(new StaticFileOptions() { FileProvider = new PhysicalFileProvider(fileManagerPath) });});app.UseMigrationsEndPoint();if (app.Environment.IsProduction()){ app.UseExceptionHandler("/Error"); app.UseHsts();}app.UseHttpsRedirection();app.UseStaticFiles();app.UseSwagger();app.UseSwaggerUI(c =>{ c.SwaggerEndpoint("/swagger/v2/swagger.json", "ServiceDeck API V2");});app.UseRouting();app.UseHangfireDashboard("/hangfire", new DashboardOptions { StatsPollingInterval = 30000});app.UseHangfireServer();app.UseAuthentication();app.UseAuthorization();app.MapControllers();app.MapBlazorHub();app.MapFallbackToPage("/_Host");app.UseHangfireDashboard();app.UseCors();app.UseEndpoints(endpoints =>{ endpoints.MapHangfireDashboard(); endpoints.MapHub<ChatHub>(ChatHub.Url); endpoints.MapHub<ReminderHub>(ReminderHub.Url); endpoints.MapHub<SubscriptionHub>(SubscriptionHub.Url); endpoints.MapHub<StripeInvoiceHub>(StripeInvoiceHub.Url); endpoints.MapRazorPages();});var backgroundJobsCronConfig = new BackgroundJobsCronConfig();builder.Configuration.GetSection(BackgroundJobsCronConfig.Position).Bind(backgroundJobsCronConfig);if (app.Environment.IsProduction()){ app.UseExceptionHandler("/Error"); app.UseHsts();}if (backgroundJobsCronConfig.Enabled){ RecurringJob.AddOrUpdate<DeleteUserAccountJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.DeleteUserAccountJob); RecurringJob.AddOrUpdate<SendTodayLeadsJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.SendTodayLeadsJob); RecurringJob.AddOrUpdate<FreeTrialEndJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.FreeTrialEndJob); RecurringJob.AddOrUpdate<ScheduledPaymentsReminderJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.ScheduledPaymentsReminderJob); RecurringJob.AddOrUpdate<QuotesExpirationCheckJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.QuotesExpirationCheckJob); RecurringJob.AddOrUpdate<DisableScheduledUsersJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.DisableScheduledUsersJob); RecurringJob.AddOrUpdate<ChatNotificationSentJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.ChatNotificationSentJob); RecurringJob.AddOrUpdate<ConvertStripeCustomerCurrencyJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.ConvertStripeCustomerCurrencyJob); RecurringJob.AddOrUpdate<ConvertServiceProviderToStripeJob>(job => job.DoWork(CancellationToken.None, null, null), backgroundJobsCronConfig.ConvertServiceProviderToStripeJob);}app.Run();// this GetEdmModel is also here solely for referenceIEdmModel GetEdmModel(){ var builder = new ODataConventionModelBuilder { Namespace = "Actions" }; builder.EnableLowerCamelCase(); var assemblyTypes = typeof(BaseController).Assembly.GetTypes(); var setTypes = assemblyTypes.Where(c => typeof(BaseController).IsAssignableFrom(c) && !c.IsGenericType && c != typeof(BaseController)).ToList(); foreach (var type in setTypes) { var genericArguments = type.BaseType.GetGenericArguments(); if (genericArguments.Length == 0) continue; var entityType = type.BaseType.GetGenericArguments()[0]; var entityTypeConfiguration = builder.AddEntityType(entityType); var setConfiguration = builder.AddEntitySet(entityType.Name, entityTypeConfiguration); var methods = type.GetMethods().Where(c => Attribute.IsDefined(c, typeof(CustomActionAttribute))); foreach (var method in methods) { dynamic target = builder.GetType().GetMethod(nameof(builder.EntityType)).MakeGenericMethod(entityType) .Invoke(builder, null); if (method.GetParameters().All(c => c.Name != "key")) target = target.Collection; var funcOrAction = Attribute.IsDefined(method, typeof(HttpGetAttribute)) ? "Function" : "Action"; dynamic funcOrActionConfig = target.GetType().GetMethod(funcOrAction).Invoke(target, new object[] { method.Name }); if (method.ReturnType == entityType) funcOrActionConfig.GetType().GetMethod("ReturnsFromEntitySet").MakeGenericMethod(method.ReturnType) .Invoke(funcOrActionConfig, new object[] { entityType.Name }); else { if (method.ReturnType != typeof(void)) funcOrActionConfig.Returns(method.ReturnType); } var parameters = Attribute.GetCustomAttributes(method, typeof(CustomActionParameterAttribute)) .Cast<CustomActionParameterAttribute>(); foreach (var par in parameters) { if (typeof(IEnumerable).IsAssignableFrom(par.Type) && par.Type != typeof(string)) { var elementType = par.Type.IsGenericType ? par.Type.GenericTypeArguments[0] : par.Type.GetElementType(); funcOrActionConfig.GetType().GetMethod("CollectionParameter").MakeGenericMethod(elementType) .Invoke(funcOrActionConfig, new object[] { par.Name }); } else funcOrActionConfig.Parameter(par.Type, par.Name); } } } return builder.GetEdmModel();}