Benchmarking Task

A Benchmarking Task is a system evaluation task with well-defined performance metrics and a set of comparable entities.

• Context:
  • It can be include a Benchmark Dataset.
  • It can range from being an Individual Benchmarking Task to being a Collective Benchmarking Task (such as organizational benchmarking).
  • It can range from being a Real-World Benchmark Task to being a Synthetic Benchmark Task.
  • It can (often) provide a standardized Assessment Framework.
  • It can foster advances and improvements by setting up a competitive environment.
  • …
• Example(s):
  • an AI Benchmarking Task, such as:
  • a Best-In-Class Benchmarking Task, such as:
  • a Business Process Benchmarking Task,
  • a Computing System Benchmarking Task such as:
    • a Machine Learning Benchmark Task;
    • a Natural Language Processing Benchmark Task;
    • a Information Retrieval Benchmark Task;
    • a Database Management Systems Benchmark Task;
    • a Supercomputing Benchmark Task;
    • a Predictive Modeling Benchmark Task,
  • a Energy Benchmarking Task,
  • a Functional Benchmarking Task,
  • a Financial Benchmarking Task,
  • an Investment Benchmarking Task,
  • an Operational Benchmarking Task,
  • an Organizational Benchmarking Task such as: a Municipal Government Benchmarking Task; a Hospital Benchmarking Task,
  • a Performance Benchmarking Task,
  • a Project Management Benchmarking Task,
  • a Product Benchmarking Task,
  • a Strategic Benchmarking Task.
  • a Clinical Trial Benchmarking Task.
  • …
• Counter-Example(s):
  • a Two-Player Game.
  • a Theoretical Analysis.
• See: Performance Metric, Performance Indicator, Strategic Management, Best Practice.

---

---

References

2019a

• (Wikipedia, 2019) ⇒ https://en.wikipedia.org/wiki/Benchmarking Retrieved:2019-11-10.
  • Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.

    Benchmarking is used to measure performance using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of performance that is then compared to others. Also referred to as "best practice benchmarking" or "process benchmarking", this process is used in management in which organizations evaluate various aspects of their processes in relation to best-practice companies' processes, usually within a peer group defined for the purposes of comparison. This then allows organizations to develop plans on how to make improvements or adapt specific best practices, usually with the aim of increasing some aspect of performance. Benchmarking may be a one-off event, but is often treated as a continuous process in which organizations continually seek to improve their practices. In project management benchmarking can also support the selection, planning and delivery of projects. In the process of best practice benchmarking, management identifies the best firms in their industry, or in another industry where similar processes exist, and compares the results and processes of those studied (the "targets") to one's own results and processes. In this way, they learn how well the targets perform and, more importantly, the business processes that explain why these firms are successful. According to National Council on Measurement in Education, benchmark assessments are short assessments used by teachers at various times throughout the school year to monitor student progress in some area of the school curriculum. These also are known as interim assessments.

    In 1994, one of the first technical journals named Benchmarking: An International Journal was published.

2019b

• (Wikipedia, 2019) ⇒ https://en.wikipedia.org/wiki/Benchmark_(computing) Retrieved:2019-11-10.
  • In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.

    The term benchmark is also commonly utilized for the purposes of elaborately designed benchmarking programs themselves.

    Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems (DBMS).

    Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.

     Test suites are a type of system intended to assess the correctness of software.

---