- Staging API: https://staging.mathmlcloud.org
- Virtual Machine size: Standard Tier A1 (1 core, 1.75 GB RAM, see http://azure.microsoft.com/en-us/pricing/details/virtual-machines/#Linux)
- Code version deployed: 5.2.2, https://github.com/benetech/mmlc-api/tree/v5.2.2
- Test scripts: Apache JMeter v2.12, run from an OS X 10.10 workstation, Java 6.0_35
- HTML file upload through the API
- Data set of three files
- textbook-sample-math.html = 5 equations from an O'Reilly book
- asciimath.html = 10 equations from Chapman site (similar to above)
- mathmlcloud-tests.html = 110 equations from original sample prepared for MMLC testing
- Configuration 1: 10 users, each uploading 10 files from the data set (round robin selection)
- Configuration 2: 5 users, each uploading 50 files from the data set (round robin selection)
- Configuration 3: 3 users, each uploading 50 files from the data set (round robin selection)
- New users created for each test
- 15 seconds pause between each equation
- 5 minute response timeout
- Data set of three files
- The Node system was restarted before each test run
- JMeter script and data files are in GitHub repository mmlc-regression-html
|Scenario||Number of samples||Average (ms)||Median||Min||Max||Throughput|
|10 users, 10 files||100||68123||24032||1747||150729||6.2/min|
|5 users, 50 files||250||10730||594||205||157915||11.6/min|
|3 users, 50 files||150||22432||1165||217||156371||4.8/min|
- I ran each of the scenarios a couple of times, and results were not consistent for the same configuration, even though the Node system was restarted each time. Our VM configuration could sometime hold up for 100-150 file upload requests before being overloaded and exceeding the 5 minute request timeout, sometimes hold up for 200 or more file requests.
- Throughput appears to top out at about 11 file uploads/minute, but decreases over time, suggesting that the server either cannot keep up or that there is a memory leak. This value, like response time, was inconsistent across runs.
- For the first two test runs, after the test run was completed, the MMLC app had issues getting data from the API, and so appeared unresponsive. This seems to suggest an issue with memory menagement that is not allowing the system to return to a stable place.
- The console log does not show any error output, but some runs began getting timeout errors as the test went. We cannot yet see into the Node server, though, to identify bottlenecks or memory use issues.
- Scale: Our VM configuration is for a small, single CPU machine. Node.js and Sails.js scalability appears to be best done by running an instance of Node per CPU core. To get better throughput we should explore the options available for scaling Node/Sails applications, and run the tests with the system running on multiple cores.
- Monitor: To deal with stability issues, we should gather tools that would allow us to identify potential memory leaks or bottlenecks in a Node.js system.
- Repeatability: These test numbers give one view of the system, but more work needs to be done to get more consistently repeatable results.