Benchmark Configuration
The following describes the benchmark configuration, unless otherwise noted below.
- AMI : LANSA Web Server 20150216-2103 (ami-992452a3)
- Windows Server 2012 R2 with all Windows updates applied
- Multi AZ Auto Scaling Group
- SQL Server Web 11.00.2100.60.v1, Single AZ
- LANSA performance improvements in V14.
- Tested 12th March 2015 with a beta version of Visual LANSA V14.
- AWS Region ap-southeast-2 (Sydney), unless otherwise stated.
- Load test tool Loadster Workbench 3.5.6 running in 5 Regions - Virginia, Singapore, Ireland, Brazil, Sydney
- Benchmark includes select, insert, update and delete transactions with appropriate delays between transactions for each user.
- The Base Cost column provides indicative pricing. Only the EC2 instances and the RDS instance are included in the cost. There are other charges which are generally a small percentage of these costs which you will incur. You should monitor the costs closely to ensure it is within your budgetary constraints. RDS costs do not include data storage costs and, in particular, neither Provisioned IOPS increased charges. Auto scaling may also increase the number of web servers running to handle an increased load, further increasing the costs.
|
Concurrent
Users
|
Web Servers
(# EC2 Instances)
|
EC2 Instance Type
(Max CPU %)
|
RDS Instance
Type
|
SQL Server
|
Avg Response
Time (seconds)
|
Max
Response
Time
(Seconds)
|
Test #
|
Base Cost ($ per hour)
|
|
30
|
4
|
m3.medium (67%)
|
db.m3.medium
|
sqlserver-web
|
1.72
|
10.6
|
188
|
0.73
|
|
30
|
2
|
t2.micro (67%)
|
db.m3.medium
|
sqlserver-web
|
1.3
|
5.82
|
212
|
0.246
|
|
40
|
4
|
t2.micro (47%)
|
db.m3.medium
|
sqlserver-web
|
1.56
|
6.65
|
214
|
0.282
|
|
40
|
4
|
t2.small (47%)
|
db.m3.medium
|
sqlserver-web
|
1.58
|
6.81
|
285
|
0.354
|
|
|
|
|
|
|
|
|
|
|
|
80 (Single AZ)
|
10
|
m3.medium (75%)
|
db.m3.large
|
sqlserver-web
|
1.45
|
7.66
|
166b
|
1.72
|
|
80
|
10
|
m3.medium (77%)
|
db.m3.large
|
sqlserver-web
|
1.75
(20% slower than single-AZ)
|
6.86
|
185
|
1.72
|
|
80
|
5
|
t2.micro (79%)
|
db.m3.large
|
sqlserver-web
|
1.45
|
5.59
|
221
|
0.51
|
|
80
|
3
|
t2.medium (67%)
|
db.m3.large
|
sqlserver-web
|
1.5
|
122.11
|
226
|
0.636
|
|
80
|
4
|
m3.large (65%)
|
db.m3.large
|
sqlserver-web
|
1.15
|
4.93
|
238
|
1.456
|
|
80 (Single AZ)
|
10 (Virginia)
|
m3.medium (85%)
|
db.m3.large
|
sqlserver-web
|
1.34
|
14.67
|
197
|
1.72
|
|
80 (Single AZ)
|
10 (Sao Paulo)
|
m3.medium (78%)
|
db.m3.large
|
sqlserver-web
|
1.1
|
3.9
|
206
|
1.72
|
|
80 (Single AZ)
|
10 (Tokyo)
|
m3.medium (60%)
|
db.m3.large
|
sqlserver-web
|
1.3
|
6.16
|
207
|
1.72
|
|
|
|
|
|
|
|
|
|
|
|
100
|
8
|
m3.medium (90%)
|
db.m3.xlarge
|
sqlserver-web
|
1.9
|
11.47
|
151
|
1.88
|
|
600
|
60
|
m3.medium (77%)
|
db.m3.2xlarge
|
sqlserver-web
|
1.85
|
7.75
|
160b
|
9.48
|
|
|
|
|
|
|
|
|
|
|
|
100
|
10
|
m3.medium (80%)
|
db.r3.xlarge
|
sqlserver-web
|
1.5
|
7.2
|
158
|
2.18
|
|
300
|
30
|
m3.medium (70%)
|
db.r3.xlarge
|
sqlserver-web
|
2.1
|
7.59
|
164
|
4.78
|
|
300
|
10
|
t2.medium (55%)
|
db.r3.xlarge
|
sqlserver-web
|
2.05
|
7.08
|
233
|
1.6
|
|
300
|
10 (PIOPS 2000)
|
t2.medium (58%)
|
db.r3.xlarge
|
sqlserver-web
|
1.84
(10% improvement)
|
7.02
|
261
|
1.6
|
|
|
|
|
|
|
|
|
|
|
|
500
|
60
|
m3.medium (73%)
|
db.r3.2xlarge
|
sqlserver-se
|
1.65
|
7.5
|
175
|
10.82
|
|
500
|
15
|
t2.medium (75%)
|
db.r3.2xlarge
|
sqlserver-se
|
1.1
|
5.48
|
232
|
4.1
|
|
500
|
10
|
m3.xlarge (82%)
|
db.r3.2xlarge
|
sqlserver-se
|
1.2
|
5.08
|
242
|
8.2
|
|
500
|
90
|
m3.medium (56%)
|
db.r3.2xlarge
|
sqlserver-se
|
1.5
|
7.63
|
177
|
14.72
|
|
|
|
|
|
|
|
|
|
|
|
600
|
60
|
m3.medium (72%)
|
db.r3.2xlarge
|
sqlserver-se
|
2
|
8.24
|
173
|
10.82
|
|
750
|
90
|
m3.medium (56%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.25
|
14.32
|
170
|
14.72
|
|
750
|
18
|
t2.medium (68%)
|
db.r3.2xlarge
|
sqlserver-se
|
2.3
|
9.02
|
236
|
4.316
|
|
750
|
18 (PIOPS 1000)
|
t2.medium (73%)
|
db.r3.2xlarge
|
sqlserver-web
|
2.1
(10% improvement)
|
9.28
|
266
|
3.096
|
|
900
|
120
|
m3.medium (38%)
|
db.r3.2xlarge
|
sqlserver-se
|
4.75
|
21.57
|
167
|
18.62
|
|
900
|
30
|
t2.medium (40%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.5
|
11.67
|
234
|
5.18
|
|
900
|
20
|
m3.xlarge (37%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.4
|
12.29
|
241
|
13.38
|
|
900
|
10
|
m3.2xlarge (38%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.45
|
19.47
|
245
|
13.38
|
|
900
|
10
|
c3.2xlarge (31%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.45
|
15.98
|
246
|
10.54
|
|
900
|
10
|
c4.2xlarge (25%)
|
db.r3.2xlarge
|
sqlserver-se
|
3.45
|
17.41
|
247
|
10.75
|
|
Note that little difference was seen between db.m3.large and db.m3.xlarge for the benchmark. So this would indicate that it's not worth spending extra for db.m3.xlarge. It is unclear as to why this is so and why moving from db.m3.large to db.m3.2xlarge which is 4 times the price yet gives 6 times the performance. If the instance types are compared, apart from doubling CPU and memory for each step, db.m3.2xlarge has High network performance. So it seems that this is the factor which allows the dramatic improvement in throughput for db.m3.2xlarge. Of course this comment is purely in the context of this benchmark. Another application may experience different behavior.
