Netcore API not connect to MySQL RDS on AWS Lambda

Question

I have api with netcore 3.1 in aws lambda that can't connect with mysql rds. Locally it works correctly but in aws lambda it doesn't.

I use entity framework. Scaffold-dbcontext with Pomelo.EntityFrameworkCore.Mysql.

The answer i get in postman is 504 api gateway timeout

Startup.cs:

services.AddDbContext<APIDbContext>(options => options.UseMySql("server=xxxxx.xxx.us-west-1.rds.amazonaws.com;database=db;user=admin;password=pass;"));

APIDbContext.cs:

public APIDbContext()
{
}

public APIDbContext(DbContextOptions<APIDbContext> options)
            : base(options)
{
}

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    if (!optionsBuilder.IsConfigured)
    {
        optionsBuilder.UseMySql("server=xxxxx.xxx.us-west-1.rds.amazonaws.com;database=db;user=admin;password=pass;");
    }

Testing I did the following and if it worked:

string dataString = "";

MySqlConnectionStringBuilder cnString = new MySqlConnectionStringBuilder();

cnString.Server = "xxxxx.xxx.us-west-1.rds.amazonaws.com";
cnString.UserID = "admin";
cnString.Password = "pass";
cnString.Database = "db";
cnString.Port = 3306;

using (MySqlConnection conn = new MySqlConnection(cnString.ToString()))
{
    conn.Open();

    MySqlCommand cmd = new MySqlCommand("SELECT Office FROM offices LIMIT 1", conn);

    using (MySqlDataReader dr = cmd.ExecuteReader())
    {
        while (dr.Read())
        {
            dataString = dr["Office"].ToString();
        }
    }
}

return Ok(dataString);

file.csproj

<ItemGroup>
    <PackageReference Include="AWS.Logger.AspNetCore" Version="2.2.0" />
    <PackageReference Include="AWSSDK.SecretsManager" Version="3.3.103.11" />
    <PackageReference Include="Microsoft.NETCore.Targets" Version="3.1.0" PrivateAssets="all" />
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.1.3" />
    <PackageReference Include="AutoMapper" Version="10.0.0" />
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="8.0.1" />
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.3.101" />
    <PackageReference Include="AWSSDK.S3" Version="3.3.111.37" />
    <PackageReference Include="HtmlSanitizer" Version="5.0.343" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.6" />
    <PackageReference Include="Microsoft.AspNetCore.Cors" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.2.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="3.1.6" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="3.1.6">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="3.1.6" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="3.1.6">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="3.1.6" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="3.1.6" />
    <PackageReference Include="Microsoft.IdentityModel.Clients.ActiveDirectory" Version="5.2.8" />
    <PackageReference Include="Microsoft.OData.Client" Version="7.7.0" />
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="3.1.4" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="3.1.2" />
    <PackageReference Include="Pomelo.EntityFrameworkCore.MySql.Design" Version="1.1.2" />
    <PackageReference Include="RestSharp" Version="106.11.4" />
    <PackageReference Include="SendGrid" Version="9.20.0" />
    <PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />
    <PackageReference Include="Serilog.Enrichers.Environment" Version="2.1.3" />
    <PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
    <PackageReference Include="Serilog.Settings.Configuration" Version="3.1.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
    <PackageReference Include="Serilog.Sinks.File" Version="4.1.0" />
    <PackageReference Include="Serilog.Sinks.RollingFile" Version="3.3.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="5.5.1" />
    <PackageReference Include="Swashbuckle.AspNetCore.Newtonsoft" Version="5.5.1" />
    <PackageReference Include="Z.EntityFramework.Plus.EFCore" Version="3.0.57" />
  </ItemGroup>

Why doesn't it work with the settings I have in startup.cs and apidbcontext.cs?

Thank you very much for the help.

Update:

Following @lauxjpn recommendation to print the connection string, I read it like this:

    public class OfficesCatalogsController : ControllerBase
    {
        private readonly APIDbContext _context;
        private readonly ILogger<OfficesCatalogsController> _logger;

        public OfficesCatalogsController(APIDbContext context, ILogger<OfficesCatalogsController> logger)
        {
            _context = context;
            _logger = logger;
        }

    
        [HttpGet]
        public async Task<IActionResult> GetAllOffices()
        {
        
        try
            {

                _logger.LogInformation("Logeando ando inicio try");

                var cxn = _context.Database.GetDbConnection().ConnectionString;

                _logger.LogInformation("ConnString: " + cxn);
                
                
                var query = (from office in _context.Officescatalog
                             select new
                             {
                                 office.IdOffice,
                                 office.Office
                             });

                var result = await query.ToListAsync();

                if (result == null) return NotFound();

                _logger.LogInformation("Logeando ando fin try");
                return Ok(result);
            }
            catch (Exception ex)
            {
                _logger.LogInformation("Catch");
                _logger.LogError(ex.Message, ex.StackTrace);
                return new JsonResult(ex.Message) { StatusCode = (int)HttpStatusCode.InternalServerError };
            }
        }
    }

In cloudwatch I never see the connection string print out, but if I run it on my local machine and connection string to the rds db, it works perfectly

Cloudwatch logs:

2020-08-08T10:52:32.828-05:00
START RequestId: ce1e574e-efc1-42f5-8da1-76a834ee Version: $LATEST

2020-08-08T10:52:32.828-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository[50]

2020-08-08T10:52:32.829-05:00
Using an in-memory repository. Keys will not be persisted to storage.

2020-08-08T10:52:32.829-05:00
[Warning] Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository: Using an in-memory repository. Keys will not be persisted to storage.

2020-08-08T10:52:32.829-05:00
[Warning] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

2020-08-08T10:52:32.829-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[59]

2020-08-08T10:52:32.838-05:00
Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

2020-08-08T10:52:32.838-05:00
[40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]

2020-08-08T10:52:32.839-05:00
Creating key {7875e158-x} with creation date 2020-08-08 15:52:32Z, activation date 2020-08-08 15:52:32Z, and expiration date 2020-11-06 15:52:32Z.

2020-08-08T10:52:32.858-05:00
[Information] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Creating key {7875e158-x} with creation date 2020-08-08 15:52:32Z, activation date 2020-08-08 15:52:32Z, and expiration date 2020-11-06 15:52:32Z.

2020-08-08T10:52:32.858-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]

2020-08-08T10:52:32.859-05:00
No XML encryptor configured. Key {7875e158-x} may be persisted to storage in unencrypted form.

2020-08-08T10:52:33.367-05:00
[Warning] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: No XML encryptor configured. Key {7875e158-x} may be persisted to storage in unencrypted form.

2020-08-08T10:52:33.367-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:52:33.368-05:00
Application started. Press Ctrl+C to shut down.

2020-08-08T10:52:33.377-05:00
[Information] Microsoft.Hosting.Lifetime: Application started. Press Ctrl+C to shut down.

2020-08-08T10:52:33.377-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:52:33.377-05:00
Hosting environment: Production

2020-08-08T10:52:33.377-05:00
[Information] Microsoft.Hosting.Lifetime: Hosting environment: Production

2020-08-08T10:52:33.378-05:00
[Information] Microsoft.Hosting.Lifetime: Content root path: /var/task

2020-08-08T10:52:33.378-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:52:33.378-05:00
Content root path: /var/task

2020-08-08T10:52:33.728-05:00
[40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]

2020-08-08T10:52:33.728-05:00
Request starting GET https://xxxxxxxxx.execute-api.us-west-1.amazonaws.com/Prod/offices 0

2020-08-08T10:52:33.766-05:00
[Information] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting GET https://xxxxxxxxx.execute-api.us-west-1.amazonaws.com/Prod/offices 0

2020-08-08T10:52:34.548-05:00
[40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]

2020-08-08T10:52:34.548-05:00
Executing endpoint 'API.Controllers.OfficesCatalogsController.GetAllOffices (PI)'

2020-08-08T10:52:34.566-05:00
[Information] Microsoft.AspNetCore.Routing.EndpointMiddleware: Executing endpoint 'API.Controllers.OfficesCatalogsController.GetAllOffices (API)'

2020-08-08T10:52:34.807-05:00
[40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]

2020-08-08T10:52:34.807-05:00
Route matched with {action = "GetAllOffices", controller = "OfficesCatalogs"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] GetAllOffices() on controller API.Controllers.OfficesCatalogsController (API).

2020-08-08T10:52:34.808-05:00
[Information] Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker: Route matched with {action = "GetAllOffices", controller = "OfficesCatalogs"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] GetAllOffices() on controller API.Controllers.OfficesCatalogsController (API).

2020-08-08T10:52:37.568-05:00
[40m[32minfo[39m[22m[49m: API.Controllers.OfficesCatalogsController[0]

2020-08-08T10:52:37.568-05:00
Logeando ando

2020-08-08T10:52:37.585-05:00
[Information] API.Controllers.OfficesCatalogsController: Logeando ando

2020-08-08T10:52:37.585-05:00
[Information] API.Controllers.OfficesCatalogsController: Logeando ando inicio try

2020-08-08T10:52:37.585-05:00
[40m[32minfo[39m[22m[49m: API.Controllers.OfficesCatalogsController[0]

2020-08-08T10:52:37.585-05:00
Logeando ando inicio try

2020-08-08T10:53:03.425-05:00
END RequestId: ce1e574e-efc1-42f5-8da1-76a834ee

2020-08-08T10:53:03.425-05:00
REPORT RequestId: ce1e574e-efc1-42f5-8da1-76a834ee Duration: 30028.08 ms Billed Duration: 30000 ms Memory Size: 256 MB Max Memory Used: 256 MB Init Duration: 1865.88 ms

2020-08-08T10:53:03.425-05:00
2020-08-08T15:53:03.425Z ce1e574e-efc1-42f5-8da1-76a834ee Task timed out after 30.03 seconds

2020-08-08T10:53:04.869-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository[50]

2020-08-08T10:53:04.869-05:00
Using an in-memory repository. Keys will not be persisted to storage.

2020-08-08T10:53:04.870-05:00
[Warning] Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository: Using an in-memory repository. Keys will not be persisted to storage.

2020-08-08T10:53:04.871-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[59]

2020-08-08T10:53:04.871-05:00
Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

2020-08-08T10:53:04.871-05:00
[Warning] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

2020-08-08T10:53:04.879-05:00
[40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]

2020-08-08T10:53:04.879-05:00
Creating key {a531edfd-x} with creation date 2020-08-08 15:53:04Z, activation date 2020-08-08 15:53:04Z, and expiration date 2020-11-06 15:53:04Z.

2020-08-08T10:53:04.880-05:00
[Information] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Creating key {a531edfd-x} with creation date 2020-08-08 15:53:04Z, activation date 2020-08-08 15:53:04Z, and expiration date 2020-11-06 15:53:04Z.

2020-08-08T10:53:04.896-05:00
[40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]

2020-08-08T10:53:04.896-05:00
No XML encryptor configured. Key {a531edf-x} may be persisted to storage in unencrypted form.

2020-08-08T10:53:04.897-05:00
[Warning] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: No XML encryptor configured. Key {a531edfd-x} may be persisted to storage in unencrypted form.

2020-08-08T10:53:05.359-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:53:05.359-05:00
Application started. Press Ctrl+C to shut down.

2020-08-08T10:53:05.360-05:00
[Information] Microsoft.Hosting.Lifetime: Application started. Press Ctrl+C to shut down.

2020-08-08T10:53:05.368-05:00
[Information] Microsoft.Hosting.Lifetime: Hosting environment: Production

2020-08-08T10:53:05.368-05:00
[Information] Microsoft.Hosting.Lifetime: Content root path: /var/task

2020-08-08T10:53:05.368-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:53:05.368-05:00
Hosting environment: Production

2020-08-08T10:53:05.368-05:00
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]

2020-08-08T10:53:05.368-05:00
Content root path: /var/task

LambdaEntryPoint.cs

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;

namespace API
{

    public class LambdaEntryPoint :

    
        Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction
    {
  
        protected override void Init(IWebHostBuilder builder)
        {
            builder
                .UseStartup<Startup>();
        }

     
        protected override void Init(IHostBuilder builder)
        {
        }
    }
}

LocalEntryPoint.cs

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;

namespace API
{

public class LocalEntryPoint
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });
    }
}

Will I have to configure something from the lambda entry point or something similar for the connection to be loaded in aws lambda?

Answer 1

First fix your logging, so that the connection string actually gets output to the log. See Logging in .NET Core and ASP.NET Core: Configure logging for how to setup your appsettings.json (or appsettings.Production.json ) file to accomplish that.

It should look similar to this:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information",

      "Microsoft.EntityFrameworkCore" : "Debug",
      "YourNamespace.OfficesCatalogsController": "Debug"
    }
  }
}

---

Update

Actually, the following line might open the connection, which then timeouts, which is why you never see the connection string in the log:

var cxn = _context.Database.GetDbConnection().ConnectionString;

To definitely get the actual connection string, you could use an interceptor:

public class ConnectionInterceptor : DbConnectionInterceptor
{
    public override InterceptionResult ConnectionOpening(DbConnection connection, ConnectionEventData eventData, InterceptionResult result)
    {
        // TODO:
        // Log the connection string here.
        
        return base.ConnectionOpening(connection, eventData, result);
    }

    public override Task<InterceptionResult> ConnectionOpeningAsync(DbConnection connection, ConnectionEventData eventData, InterceptionResult result, CancellationToken cancellationToken = new CancellationToken())
    {
        // TODO:
        // Log the connection string here.

        return base.ConnectionOpeningAsync(connection, eventData, result, cancellationToken);
    }
}

You can configure it like this:

services.AddDbContext<APIDbContext>(
    options => options
        .UseMySql("server=xxxxx.xxx.us-west-1.rds.amazonaws.com;database=db;user=admin;password=pass;")
        .AddInterceptors(new ConnectionInterceptor()));

---

Update

It is also possible, that if you have connection pooling enabled (which by default keeps 100 connections per same connection string opened), that you could exceed the max connections setup for the RDS instance.

To test this, add Pooling=false to your connection string and restart your RDS instance.

If you can suddenly connect without issues after that, then this behavior might have been the issue.