Services Coverage

Measurement Types

• Availability is measured including not only HTTP problems and “time out” but also critical interoperability issues and “sorry” error messages detection.
  
  
• Performance is calculated with breakdown by network attachment, PDP context, threads, frame rates, etc.
  
  
• Usability issues are detected: legal, deontological and commercial requirements, navigation, user interface, content freshness
  
  
• Interoperability with content is checked for 1000+ handsets

Media

• WAP 1.x, 2.x, xHTML, imode
• SMS P2P, SMS+
• MMS P2P, MMS+
• Email
• SyncML
• Ringtones, Full Track Music Download
• Logos, Wallpapers
• Video Download / Mobile TV
• Java Games and Applications
• HTTP, USSD, IVR

• GSM, CDM, GPRS, EDGE, UMTS, HSDPA, 1xRTT, EV-DO etc

• 60 Manufacturers
• 1200+ Handsets
• 10000+ User Agents

Testing Lifecycle: Support from Beginning to End

In order to improve the real end-user experience, quality management needs to be a constant and on-going process. Thus the entire service life cycle has to be covered.

The test resources are pooled & geographically distributed: SIMs, modems, robots.

Test cases & test campaigns repository are common and reusable.

Measurements are comparable within internal departments (i.e. marketing / network) and/or third party companies (i.e. SLA between carriers and content providers).

Unit Testing Phase

Test Builder is used by developers to test & troubleshoot a service iteratively, handset by handset with the ability to use wireless modems and to get immediate results as: page look & feel issues, full resources description, interoperability issues, source code validation…

Acceptance Testing Phase

During pre-production phase QA managers check overnight thousands of pages from their service against dozens of handsets until final acceptance.
During production phase acceptance tests are rerun every time a new handset appears on the market or whenever a network component changes.
Specific reports corresponding to this phase aggregate:

• Technical issues: broken links/pictures, unavailable services, etc.
• Optimisation issues: source code problems/optimization, content compliance, resources size/format optimization (pages, pictures…), multipart mime optimisation,
• Billing issues: billing transaction problems, provisioning mechanisms (subscribe / unsubscribe), etc.
• Interoperability issues: immediate detection of handset interoperability issues
• Guidelines compliance

Live Monitoring & Alerting Phase

When a service goes into production, operations engineers schedule hi-frequency user based scenarios from different locations against a limited number of typical popular handsets.

Availability (including interoperability issues and user friendly error messages) and performance are compared to targets in real time; alarms are raised through SNMP, SMS or email. A test run can be reopened in Test Viewer for deep troubleshooting.

Performance KPIs include: first byte received, breakdown on authentication, dialling, network register, PDP context, IP frames,...

QoS & SLA Management Phase

Different measurements are aggregated in various daily, weekly, monthly reports showing business and technical KPIs in order to:

• Manage SLAs of MVNOs and Third Parties
• Trend analysis for network & service dimensioning
• Breakdown time of home page access between network components
• Benchmark handsets against services, etc.
• Detection of unauthorized adult content, content freshness verification, conformity to guidelines, ergonomics, presentation…

Customer Intelligence Phase

Measurements are correlated and aggregated with other measurements coming from other sources: OSS, passive probes, billing database, Mercury Test Director, etc.

This complex analysis can be used to provide marketing and other executives with information relevant to their business decision-making, as for example, comparative information about lost revenues or performance comparisons with competitors.