Tools Comparison
| Tool Name | Key Features | Pricing | Uniqueness |
| LoadRunner | • No need to install it on the server under test. It uses native monitors. For Ex: perfmon for windows or rstatd daemon for Unix • Uses ANSI C as the default programming language1 and other languages like Java and VB. • Excellent monitoring and analysis interface where you can see reports in easy to understand colored charts and graphics. • Supports most of the protocols. • Makes correlation3 much easier. We will dig into correlation through a series of posts later. • Nice GUI generated script through a one click recording; of course you would need to modify the script according to your needs. • Excellent tutorials, exhaustive documentation and active tool support from HP. | • Free (50 concurrent users) • Unleashed: Contact vendor for pricing | • Loadrunner allows you to see test results in real-time or easily compare multiple tests, all with a simple to use interface and intuitive graphs • It has flexible pricing that accommodates hourly, monthly or annual usage. No setup or maintenance fees • Loadrunner’s professional services team can help you to set up and run your performance tests |
| Gatling | • Standalone HTTP Proxy Recorder • Scala-based scripting • An expressive self-explanatory DSL for test development • Asynchronous non-blocking engine for maximum performance • Excellent support of HTTP(S) protocols and can also be used for JDBC and JMS load testing • Validations and assertions • Comprehensive HTML Report | • Open Source version is free • Frontline On-premises: Unlimited testing, starting at 400€/mo • Frontline Cloud/based: Unlimited testing, starting at 3€/hour | • Gatling is written in Scala, which allows you to run it on any system. This also means you won’t hit any trouble when using different local machines and cloud servers to run and create your tests. • It has a metrics dashboard that you can see after test execution without having to add any additional plugins. The report is stored as an HTML file, which can be easily saved for later analyses. • It has an integrated assertions API, which gives you a full arsenal to perform functional assertions. This API allows you to run different types of functional checks along with your performance testing. • Tests are very elegant because Gatling defines a domain-specific language that allows writing very clear and human-readable tests. |
| JMeter | • User-friendly GUI: JMeter comes with a simple and interactive GUI • Platform independent: JMeter is written and developed using java, so it can run on any environment or workstation that accepts a Java virtual machine, for example — Windows, Linux, Mac, etc. • Supports various server types: Web (HTTP, HTTPS, SOAP), database (JDBC, LDAP, JMS) and mail (POP3). • Supports multiple protocols such as HTTP, JDBC, LDAP, SOAP, JMS, and FTP. • Simulates multiple users by using virtual users or unique users in order to generate heavy load against web application under test. • A multithreading framework allows concurrent and simultaneous sampling of different functions by many or separate thread groups. • Remote distributed testing: JMeter uses a Master-Slave concept for distributed testing where the master will distribute tests among all the slaves and the slaves will execute scripts against your server. • Test results can be viewed in different formats like graph, table, tree, and report, etc. | Free / Open Source | • JMeter has a very large community around it with endless resources and integrates with almost every performance testing tool on the market • Easy installation: It can be installed on any desktop with Windows, Mac or Linux • The test IDE allows test recording from browsers or native applications • Readily available plugins, for example, visualization plugin for data analysis |
| Blazemeter | • 100% Compatible with Apache JMeter • Create tests on any scale – up to 1 million concurrent users • Set up tests within minutes • Run scriptless tests or upload URL lists • Run from the cloud or on-premise • Run tests from multiple geo-locations • Simulate mobile testing from real devices • Facilitates team collaboration with script and report sharing • Multi-tool support • Integrations with leading CI and APM tools • Support and professional services • Real-time, detailed reporting of load test results • Set KPIs for tested app behavior • Desktop and mobile user experience monitoring | • Free (50 concurrent users) • Basic: $99/mo (1000 concurrent users) • Pro: $499/mo (5000 concurrent users) • Unleashed: Contact vendor for pricing | • BlazeMeter allows you to see test results in real-time or easily compare multiple tests, all with a simple to use interface and intuitive graphs • It has flexible pricing that accommodates hourly, monthly or annual usage. No setup or maintenance fees • BlazeMeter’s professional services team can help you to set up and run your performance tests |
| Agileload | • The intuitive graphic interface lets you capture and design sophisticated business scenarios. • Automatic detection and analysis of performance issues. • Highly customizable scripting and reporting engine. • Support performance testing across the application lifecycle. • Test against a broad range of applications and protocols. • Record and replay RIAs using a variety of Web 2.0 technologies. • Measurable and repeatable load generation with system and end-user monitoring. • Automatic recording of test scenarios. • Simulate a realistic user load with an Agileload load injection engine. • Runs under Microsoft Windows (2003 and higher) and is compatible with all web browsers. | • Free version: Build all your test scenarios with 10 virtual users. • Pay as you go: Power up your Agileload by renting from 50 to 10000 virtual users. | • The free version or “starter kit” allows for quick ramp-up use of the tool. • Automatic modeling vastly reduces scripting time for subsequent test cases against the same app. • End-to-end diagnostic features: The tool not only monitors the front-end but also the back-end to detect the reason for performance degradations. • Anomaly detection and suggestion features help quickly identify and resolve performance bottlenecks. • Configurable reports, for a mix of audiences. |
| Redline13 | • Build packaged, custom, and repeatable tests that can easily be delivered to QA, streamlining your testing process. • Take complete control of your tests by running them in your own AWS environment. The benefits of the cloud, with the controls of on-premise. • All RedLine13 user’s load tests are backed up and saved for your convenience. When you need to run the same tests again, simply hit the “clone test” button. • Run an unlimited amount of tests on an unlimited amount of servers for as long as you’d like. • Scale-out your tests from 1,000 users even up to 1,000,000 users. • Access our own open-source load testing plugins, craft your own, or use one from UBIK to streamline your testing process • Integrates with Jenkins to complete the continuous integration process for load testing • From real-time graphs to generating downloadable reports, constantly monitor your load tests with its reporting features | • Free plan • The basic plan for a single user: $75/mo; $500/year • Premium plan for a testing team: $299/mo; $2,500/year • Custom enterprise plan: Contact the vendor for pricing information | • Open architecture for building and running load tests • Allows you to build packaged, custom, and repeatable tests • It allows you to take complete control of your tests by running them in your own AWS environment. The benefits of the cloud, with the controls of on-premise. |
| Neoload | • Broad technology support • Scriptless design and visual programming • Live debugging and monitoring • High load support • Network virtualization • Device/Browser Simulator • Cloud platform • Broad infrastructure monitoring support • Detailed reports and in-depth results analysis • It’s easy to collaborate as you can share test results, designs, and a pool of virtual users • Offers several integrations and APIs | • Free: For developers and testers running small load or performance tests • Standard: For one tester testing one application at a time • Professional: For teams testing one application at a time • Enterprise: For organizations testing multiple applications concurrently • VUH: For organizations needing test size and licensing flexibility | • Supports performance testing of all standard and rich applications (RIA) for desktop and mobile devices • Monitors all major servers available on the market • Enterprise-class: Efficient and infinitely scalable, infrastructure monitoring capability, APM tool integration • Easy-to-use load testing tool: Scriptless test creation, GUI-driven design, automated parameterization • Low cost licensing and maintenance, flexible licensing options |
| New Relic | • Easy-to-set-up real-time instrumentation and analytics • Flexible instrumentation and dashboarding • Correlates application performance to the end-user experience • Connects application and infrastructure performance • Rich, detailed transaction data • Real-time error analysis with on-demand diagnostic tools • Integration with DevOps tooling • Cloud-service instrumentation • Built to scale | Price is based on the size of your instance, the number of instances, and the total time running. You can estimate your price at official site | • Works with all the different web development languages, making compatibility a non-issue • Works as a service so that you can access it from anywhere and anytime • Ready-to-go performance metrics dashboard allows you to drill down on the “browser page load time” to the amount of time the web application, network, DOM processing, page rendering, etc., taken at various instances of time |
| AppDynamics | • Code-level visibility • Dynamic baselining and powerful alerting • Application performance management agents • Data retention • Monitoring of multiple platforms • Troubleshoot performance issues in production • Trend database performance over time • Continuously monitor in a high-volume environment • Install and begin using immediately • Database agents • Server Visibility • Synthetic Monitoring • Mobile Real-User Monitoring • Browser Real-User Monitoring | There are three tiers with difference Licensing cost 1. APM Pro 2. APM Advanced 3.APM Peak | • Ensures that businesses get useful information with its unified analytics functionality. • AppDynamics features an innovative approach called Unified Monitoring that incorporates various AppDynamics applications into a single robust solution that helps to determine the precise origin of any performance issue |
| Dynatrace | • Full-stack discovery • Full-stack performance management • Visualizing application infrastructure • AI-powered analytics • Self-healing at scale • Detecting vMotion events • Digital experience management • Container, cloud, and infrastructure management | Dynatrace offers a free trial and difference licensing models | • Dynatrace provides big data analytics • AI-optimized to find answers and actionable insights for DevOps teams • Can conduct automated monitoring of up to more than 100,000 hosts |