Scott Gress
556b79e2d2
for #24829 See https://github.com/axios/axios/issues/1045 -- by default Axios buffers uploaded files into memory fully, to support redirects. For large file uploads this means we get OOM errors, especially when sending to multiple teams. There's a few other optimizations we can put in place here but in the short term we can fix the buffering issue by setting `maxRedirects: 0` on the requests. I tested this by adding an `onUploadProgress` handler to the Axios request that dumps memory usage, and uploading a 209mb software file to 3 teams. Before the update, the readout ticket up continuously (the first number is the # of bytes uploaded): ``` 1540129 {rss: 161652736, heapTotal: 65880064, heapUsed: 55625552, external: 28411157, arrayBuffers: 24338844} edit-software.js:177 1554313 {rss: 149254144, heapTotal: 65880064, heapUsed: 52445200, external: 25193635, arrayBuffers: 21121327} edit-software.js:177 2339833 {rss: 151703552, heapTotal: 66404352, heapUsed: 52269280, external: 12664377, arrayBuffers: 8592064} ...a minute later... 192708641 {rss: 619323392, heapTotal: 95240192, heapUsed: 55320960, external: 618952429, arrayBuffers: 614879965} edit-software.js:177 201523233 {rss: 634613760, heapTotal: 95240192, heapUsed: 58514992, external: 636581613, arrayBuffers: 632509154} edit-software.js:177 209326677 {rss: 637399040, heapTotal: 95240192, heapUsed: 56800016, external: 639441633, arrayBuffers: 635369173} ``` so we start at ~161mb, and by the time we're done, we're using 637mb of RAM. Render's free tier has a 250mb limit on apps. With `maxRedirects: 0`, we see: ``` 2669337 {rss: 151846912, heapTotal: 66404352, heapUsed: 53297400, external: 26446868, arrayBuffers: 22374419} edit-software.js:177 2279929 {rss: 152641536, heapTotal: 66404352, heapUsed: 53453664, external: 27233300, arrayBuffers: 23160851} edit-software.js:177 2228585 {rss: 153038848, heapTotal: 66404352, heapUsed: 53537096, external: 27626516, arrayBuffers: 23554067} ...a minute later... 209326677 {rss: 146989056, heapTotal: 92094464, heapUsed: 53802856, external: 14617518, arrayBuffers: 10545071} edit-software.js:177 209326677 {rss: 153051136, heapTotal: 92094464, heapUsed: 55376336, external: 22447478, arrayBuffers: 18375026} edit-software.js:177 209326677 {rss: 152129536, heapTotal: 92094464, heapUsed: 51857632, external: 22447478, arrayBuffers: 16540013} ``` showing that we start and finish with around the same amount of RAM used.
Bulk operations dashboard
A dashboard to easily manage profiles and scripts across multiple teams on a Fleet instance.
Dependencies
-
A datastore, this app was built using Postgres, but you can use a database of your choice.
-
A Redis database - For session storage.
Configuration
This app has two required custom configuration values:
sails.config.custom.fleetBaseUrl: The full URL of your Fleet instance. (e.g., https://fleet.example.com)sails.config.custom.fleetApiToken: An API token for an API-only user on your Fleet instance.
Required configuration for software features
If you are using this app to manage software across multiple teams on a Fleet instance, five additional configuration values are required:
sails.config.uploads.bucketThe name of an AWS s3 bucket where unassigned software installers will be stored.sails.config.uploads.secretThe secret for the S3 bucket where unassigned software installers will be stored.sails.config.uploads.regionThe region the AWS S3 bucket is located.sails.config.uploads.bucketWithPostfix: The name of the s3 bucket with the directory that the software installers are stored in on appended to it. If the files will be stored in the root directory of the bucket, this value should be identical to thesails.config.uploads.bucketvaluesails.config.uploads.prefixForFileDeletion: The directory path in the S3 bucket where the software installers will be stored. If the installers will be stored in the root directory, then this value can be set to ' '.
Running the bulk operations dashboard with Docker.
To run a local bulk operations dashboard with docker, you can follow these instructions.
-
Clone this repo
-
Update the following ENV variables
ee/bulk-operations-dashboard/docker-compose.ymlfile: -
sails_custom__fleetBaseUrl: The full URL of your Fleet instance. (e.g., https://fleet.example.com) -
sails_custom__fleetApiToken: An API token for an API-only user on your Fleet instance.
You can read about how to create an API-only user and get it's token here
-
Open the
ee/bulk-operations-dashboard/folder in your terminal. -
Run
docker compose up --buildto build the bulk operations dashboard's Docker image.
The first time the bulk operations dashboard starts it will Initalize the database aby running the
config/bootstrap.jsscript before the server starts.
- Once the container is done building, the bulk operations dashboard will be available at http://localhost:1337
You can login with the default admin login:
Email address:
admin@example.comPassword:
abc123