Guide:Performance Benchmarking

From STEP Modding Wiki


Info-Logo.png Notice: This is an ancillary guide to the official STEP Guide created to reduce the length of that guide. Users reading this are likely the tinkering type or are interested in more detail around configuring Skyrim in order to increase quality or performance, but this guide is not an essential component of a successful STEP installation.


Suggest first reading: Skyrim Installation Guide, Skyrim Launcher Guide

Import GPU-z log into Excel


Benchmarking for STEP

Benchmarking is work, but it is necessary and the results can be illuminating and useful. We greatly appreciate any help from our users in accumulating benchmarks according to the procedures outlined here. Other methods may be used, but the results will not be as useful a compare. Any users wishing to suggest other means that may be simpler, please do so on the Discussion page of this article. Please append '~z929669 Ixian Insignia.png Talk' to your comments.

Vanilla

  • Base vanilla game
  • All DLC
  • SKSE

... no more & no less please.

STEP:Core

  • Vanilla
  • All STEP:Core mods installed according to the STEP Guide.

STEP:Extended

  • STEP+Core
  • All optional mods in STEP installed according to the STEP Guide.

Benchmark Sequence: Opening Carriage Ride

Although there are many ways to benchmark performance in Skyrim --and many are much more comprehensive-- we recommend the opening carriage ride, because it is the only performance-intensive, script-intensive, and graphics-intensive method that is 100% reproduceable across all user configurations. Therefore, it is directly comparable:

Benchmarking Methodology

The following methods are suggested in order to maintain a high degree of comparability and reliability among benchmark runs.

  1. If mods were recently installed and IF NOT USING a SSD, defragment the game HDD. This will prevent game stutter caused by i/o inefficiencies (the effects are real!).
  2. Shut down any unnecessary applications that may be running.
  3. Launch Fraps or application to log FPS. Set to output FPS to a log file.
  4. Launch GPU-z or in order to measure GPU and system memory-usage statistics. Set these applications to log the following parameters to a log file (Sensors > Log to File; each variable log can be toggled by selecting the corresponding drop-downs):
    • GPU Core Clock (MHz)
    • GPU Temperature (°C)
    • GPU Load (%)
    • Memory Usage (Dedicated) - This is on-card-VRAM (dedicated memory usage may not be available for many video cards)
    • Memory Usage (Dynamic) - This is the amount of VRAM transferred to much-slower system RAM (dynamic memory usage may not be available for many video cards)
  5. Launch Skyrim, and start a new game. Once loaded and the sound starts and the FPS counter appears, hit the benchmark initialization hotkey (F11 by default in FRAPS).
  6. Let the game progress up to the first autosave just after jumping down off the cart, and press the FRAPS benchmark hotkey again to stop benchmarking. Total time since first depressing the benchmark hotkey will be approximately 240 seconds.
  7. Quit the game as quickly as possible and immediately exit the GPU-z app to stop logging.
  8. Go to the GPU-z log. Identify the point in the log where the GPU Core Clock first maxes out and find the point beyond which the GPU Load values and the Dedicated Memory max and become fairly consistent (it should be between 50-90 rows from the top of the file). Delete all preceding rows (keep the header row!).
  9. Move down to the end of the file and locate the point at which the GPU Load and Dedicated Memory begin to recede again (signifying loading the menue and quitting the game, usually 20-40 rows of data, depending on how much time is spent in the menu and actually shutting down the game and quitting GPU-z). Delete all rows after this point. The GPU-z log file is now trimmed of "noisy" data.
  10. There are many ways to do this, but get GPU-z log data into a spreadsheet program and calculate the mean and standard deviation for:
    • GPU Load
    • GPU Temperature
    • Memory Usage (Dedicated)
    • Memory Usage (Dynamic)
  11. Open the Fraps FPS log, and copy that column into the GPU-z spreadsheet.
  12. Download this example spreadsheet as an actual working example.
  13. As can be seen in the example spreadsheet, there are several columns and 200+ rows of data. The relevant summary statistics are the mean (i.e., average) and the standard deviation (i.e., a measure of the variation about the mean). These can be calculated for each column of data by using the following formulas in Excel or most other spreadheets (cell ranges are in parentheses):
    • = AVERAGE(A2:A240)
    • = STDEV(A2:A240)
Example benchmarks

Example benchmark results:

Kelmych
techangel85
z929669

Benchmark Run Procedure

  1. Run a vanilla benchmark for each unique user config.
  2. Run a benchmarks at each of the points graphically indicated in the STEP Guide.
    • (optional) To assess the most impactful mods, install a block of mods (a "block" is a mod Section [e.g., Fixes] or a Pack), and run a benchmark.
      1. Repeat the benchmark run to determine if the performance impact is significant (i.e., +/- 1 FPS/VRAM standard deviation difference [remember, these data can be highly volatile... and VRAM has a ceiling, so total process RAM usage may be best]).
        • IF AND ONLY IF there is a significant difference in #3, test the one or two mods that are most suspect for causing the performance hit to confirm (i.e., run vanilla + these select mods).
        • If these singular or smaller-group runs are confirmed to account for the majority of the difference, then the performance impact (+/- SD) of each should be recorded and associated with the unique user config.
        • Repeat the performance test on these single mods using the scenic carriages ride.
      2. Uninstall the previous mod "block", install another, and repeat beginning step 2.