To use data to improve services, organizations need strong processes in place to ensure data are accessible, standardized, timely, high-quality, and secure.
In this section
Data Management Processes
Data Systems
Data Quality Assurance
Data Privacy
Data Management Processes
By developing a process map to visualize how data are managed across the organization, monitoring and evaluation staff can improve organizational efficiency and accuracy in data collection and analysis. A process map provides insight into how data collection operates and can be used to increase communication and shared understanding of roles and responsibilities across a team or organization. Process mapping will identify bottlenecks, repetition, and delays, leading staff to brainstorm improvements and address challenges throughout the data management process. A data process map will also be invaluable for staff making decisions about data systems and creating data privacy and security procedures.
Lay the groundwork
- Take inventory of all data collection processes and systems in your organization, including who is responsible for them. Refer to your logic model and performance indicators to ensure you have captured all relevant data points. In your data inventory, identify or specify
- all data that are tracked, including all activities, inputs, outputs, and outcomes for all programs;
- the purpose of data collection (e.g., to generate internal or funder reports);
- the format in which data are collected;
- instruments used to collect data; and
- systems used to store and organize data.
- Gather feedback from staff and leadership on what is working well and what is creating challenges. Consider the factors that contribute to challenges: Poor or obsolete systems? Lack of staff or leadership commitment? Ambiguity around roles and responsibilities? Inefficient processes? More data collection than staff can manage?
Develop
- Consult with other staff to develop a data process map that will visually describe the flow of work and data across intake or enrollment, program participation, outputs or milestones, and outcomes or results. The map should identify tracked programs and activities; define participants, inputs, outputs, and outcomes; show who and what is involved in data processes; and reveal processes that can be improved or made more efficient. Consider the following guiding questions:
- Who collects each dataset or indicator?
- How are the data collected?
- Who enters each dataset or indicator?
- Where are the data stored?
- Who can see the data? Who analyzes the data?
- Who quality checks the data? How often are quality checks performed?
- Who reports on or discusses each dataset or indicator?
- Which data are sensitive or confidential? Are use or disclosure of the data restricted by local, federal, or internal organizational requirements?
- Clarify which staff are responsible for which data responsibilities, and add to job descriptions as necessary.
- Refer to your confidentiality and data security plans, and ensure staff understand how to handle sensitive or confidential data.
Vet
- Bring all relevant staff to the table to review the full data process map—either all at once or in relevant subgroups. Facilitate conversations about your organization’s capacity to adhere to this process.
- Are the right people assigned to the right roles?
- Do staff have enough time to collect and enter data?
- Are they comfortable with data collection technology?
- Are there staff with data analysis or data management skills who could be better positioned or trained?
- Identify if training is needed to build staff confidence and skill in data management.
Use and share
- Share your process map at team meetings during discussion of data collection or other relevant topics.
- Make the data process map part of onboarding training for all new staff.
Review
- Make your data process map a living document by updating roles and procedures to meet the needs of your work, grants reporting, personnel, and software as they change over time.
- Continue to provide or facilitate staff training and education on using data management software at regular intervals to ensure skills are kept current.
Data Systems
By completing a data process map, monitoring and evaluation staff may learn that they face organizational challenges stemming from the system or software used to house data. If a system inhibits an organization’s ability to organize and analyze data or to measure performance, it may be time to find a new system. But with so many performance management software options available—and some funder mandates—determining which software system best suits your organization’s needs can be challenging. Before picking a new system, your organizations should understand the range of options available, as well as your needs and parameters for a new or adapted system.
Lay the groundwork
- Solicit feedback about what staff, funders, and leaders need in your organization’s data systems. Reviewing your data process map (see the Data Management Processes section for detail) can help reveal gaps in your data systems.
- Learn about the data system solutions available and the benefits they provide. Options include manual and custom-built systems, mission-aligned systems, case management systems, more generalized customer relationship management systems, and dashboards or visualization tools. Consider guiding questions such as the following:
- Can the system standardize data structure, entry procedures, access, and administration across the organization?
- Can the system enhance how staff enter data? Will the system meet your organization’s needs related to device compatibility, users, automated and integrated data collection features, file upload, batch changes, form design, and data security?
- Can the system enhance how data are extracted? Does the system enable the creation of dashboards or an application programming interface (API)?
- Can the system generate reports and data visualizations, or support data analysis or special reporting requirements? How easy is it to export data from the system?
- Are you aware of all system costs, such as purchase and build costs (including potential consulting costs), maintenance costs, licensing costs, user and system administrator privileges, annual membership fees, and customer support and training costs? Have you budgeted for those costs? Are costs fixed, or will they grow if your program expands?
- Does the system offer cloud software or other built-in data security features?
- If you need to use multiple systems (e.g., because of funder requirements), can you streamline your workflows or analysis via APIs?
- Identify resources available to support your organization’s decision to adapt your current system or move to a new data system.
- What is the budget for setting up the system?
- How will you fund ongoing maintenance? Do you have internal system administration/development capacity, or do you need to contract for these services?
- What resources are available for staff time and development?
- Reach out to peer organizations, especially those that have gone through data systems migrations, for advice on data systems.
- Normalize the idea that a single data system will not solve all organizational challenges. Communicate that the goal is to select the data system that meets the most needs while minimizing challenges.
Develop
- Establish a cross-organizational committee to guide onboarding or improvement of a data system. Ideally, the committee should include your chief operating officer and staff from the information technology, development, monitoring and evaluation teams, as well as representatives from program teams.
- Create a clear list of requirements for a new or improved data system that aligns with your needs. Similarly, identify the biggest challenges or pain points in your data system or infrastructure. Identify which historical data your organization will need to migrate into a new system and what data can exist in other formats.
- Consider whether you need a new data system or whether you can upgrade a current system to meet your requirements.
- Conduct a market scan of data system vendors and products, and compare against your list of requirements. If necessary, ask vendors to provide written price estimates and demonstrate the capabilities of their systems. Cross reference with the list of requirements you need in a data system.
- Design data system architecture, and develop a plan for how the data system will be used and implemented. Once you have decided on a solution, refer to your data process map (see the Data Management Processes section), and consider the following:
- access, permissions, and user types
- opportunities for workflow automation
- data validation to improve data quality
- units of measurement (e.g., client, family, sessions, events, program)
- service tracking versus outcome tracking
- built-in reporting templates or preset queries
- level of integration across programs
- standard fields versus program-specific fields
- tracking of clients across programs
- ability to export data for analysis or presentation in other software
- Update your data process map as a diagnostic and training tool for your new or upgraded data system.
Vet
- Loop your organization’s governing board into the data system selection and implementation process and solicit their feedback, questions, and concerns.
- Pilot test system changes with a smaller group of staff members who represent a variety of functions, especially staff with data entry responsibilities. If your organization has multiple programs, consider a pilot test with a single program.
- Correct any issues encountered during pilot testing.
Use and share
- Develop training resources such as video demonstrations and self-directed step-by-step guides that are available to all staff.
- Schedule demonstrations and formal training for all staff on the new data system. Tailor training to departments as necessary, and offer training in the future as needed.
- Roll out the new data system at full scale. Multiservice organizations may consider rolling out the system on a program-by-program basis so initial problems can be resolved before full implementation.
- Use log-in reports or other methods to track which staff are using the data system and which staff are struggling to log-in. Identify staff who could use gentle reminders to use the new data system, and offer additional training and support as necessary.
- Construct feedback loops to gauge how staff feel about the new system.
Review
- Create a forum for ongoing Q&A and user feedback related to the data system. This might include a clearly identified point person for the organization, a separate “help desk” and feedback email address, a listening session series, and a “power user” in each department.
- Develop a process and a schedule for making changes to the data system after rollout.
- Create a place to catalog changes to the data system.
Data Quality Assurance
High-quality data are essential to performance management. Poor-quality data can result in wasted resources, bad decisions, and loss of credibility. But assessing the quality of your data and implementing systems to maintain high-quality data can be difficult. Data quality refers to the assessment of the information you have, relative to its purpose and its ability to serve that purpose. Data quality management (DQM) aims to maintain a high quality of information. The ultimate goal of DQM is not to create subjective notions of “high-quality” data but, rather, to increase return on investment for work that depends upon data.
Lay the groundwork
- Profile your data: Data profiling is an essential process in DQM, building on data inventories and process mapping. The process includes reviewing data in detail, comparing and contrasting the data to its own metadata, running statistical models and tests to assess quality, and reporting regularly on data quality metrics.
- Make the case for DQM to your staff and leadership: Present the benefits in the language of the program and speak to stakeholders’ critical and specific program priorities. Do not focus on data quality as an end in itself but, rather, address the components necessary to achieve program goals, operational performance, and compliance in reporting.
- Identify DQM roles: A data quality management process needs leaders. Who has these responsibilities now? Who should have them? Consider giving a high-level leader responsibility for overseeing the DQM process. Staff who work with the data will also need to be involved and understand DQM.
Develop
- Write data quality rules based on program goals and requirements.
- Embed quality controls into your data systems (e.g., adding data validation to data entry fields or building workflows such as escalations).
- Develop metrics and targets to measure and track data quality based on ACCIT principles (accuracy, consistency, completeness, integrity, and timeliness).
- Accuracy: Accurate data match reality. Accuracy should be measured through source documentation or through independent confirmation techniques.
- Consistency: Consistent data have uniform formatting and coherent values across datasets (e.g., two values pulled from separate datasets should not conflict with each other).
- Completeness: Complete data have enough information to draw necessary conclusions.
- Integrity: Data with integrity are transformed accurately and according to established procedures.
- Timeliness: Timely data are available for use when needed.
- Establish an appropriate time frame for reviewing data quality (e.g., monthly, quarterly).
- Assess the quality of your data based on quality rules and metrics.
Vet
- Review data quality rules, metrics, and initial quality assessment with staff and leadership. Get feedback on your process and possible sources of data errors.
Use and share
- Use data quality metrics when you meet with staff to express the importance of quality data.
- Repair your data by addressing issues uncovered during data profiling like missing, incomplete, or inaccurate data. Determine why, where, and how data defects originated.
- Incorporate data quality improvement goals into individual or program performance reviews.
Review
- Standardize the process for quality assurance for your data.
- Share findings of your data quality assurance management process regularly with staff and program managers. This process could be a monthly or bimonthly check by another staff member or program managers.
Data Privacy
Strong data systems require strong privacy and protections for client data. To ensure the security of private client data, monitoring and evaluation staff need to inventory data across all programs and document the security requirements of different grants. By developing a data security plan—including a full inventory of data; a designated data security officer; an access control policy; and policies on confidentiality, passwords, and reporting data security breaches—organizations can better handle confidential data. Monitoring and evaluation staff should clearly communicate data security policies to leadership and program staff and create information for clients on how their personal data are collected, stored, and used.
Lay the groundwork
- Review your data process map (see the Data Management Processes section), and update if necessary.
- Identify the staff and departments responsible for privacy and data security. The monitoring and evaluation team may have a role or support the information technology (IT) department.
- Review the necessity of collecting confidential data and personally identifiable information. Are these data required? Do they serve an important organizational need?
- Compile any existing data security regulations, policies, procedures, and protocols that are either authored by your organization or required by funders and local or federal agencies.
Develop
- Create privacy information and informed consent procedures to inform clients what personal data are being collected, how such data will be used, and whether those data can be disclosed to other entities. Define procedures for documenting consent internally.
- Develop a data security plan tailored to your organization that addresses each of the following steps listed below. Throughout the plan, give examples that pertain to your organization, so that staff know how data should be treated. Your data security plan should perform the following functions:
- Identify a data security officer and any other staff responsible for enforcing data security guidelines. Data security officers should lead the development of the data security plan.
- Create an access control policy that defines organizational roles and level of access to information systems (e.g., system admin, group admin, user).
- Identify all confidential, personally identifiable, and sensitive data currently being collected and the systems used to store these data. Identify data security requirements or regulations applicable to each of these data, including requirements for how long data need to be stored.
- Identify who needs to have access to confidential data and formulate procedures for granting, limiting, or removing access to staff.
- Develop a confidentiality pledge that is up to date with your organization’s current data security procedures.
- Identify procedures to keep confidential storage media and printouts secure at all times and to dispose of or scrub confidential storage media and printouts.
- Identify appropriate delivery mechanisms for confidential data, such as secure file transfers, password-protected files, certified mail, and hand delivery.
- Select a password policy and standard that includes who (or what software) manages passwords. Consider implementing multifactor authentication, especially for staff who have elevated privileges in your systems.
- Define procedures for managing a data security breach, including mechanisms to detect and report a breach.
- Back up important data files regularly, and establish procedures for backing up or re-creating data.
- Work with IT as needed to ensure the organization has a security training plan for staff that includes awareness of threats such as phishing and responsibilities under your organization’s policies and procedures.
- Develop a template for a data-use agreement that will enable your organization to share confidential data with other entities or receive confidential data from another source. The agreement should ensure the receiving entity takes comparable steps to protect your organization’s confidential data.
Vet
- Work with your IT team to review the data security plan with all staff and leadership. Ask them to point out any challenges in complying with the requirements.
- Address gaps in your data security plan by procuring secure data disposal software, hiring a data security officer (depending on the size of your organization), or contracting with a data security service or consultant.
Use and share
- Implement your data security plan, and ensure current processes conform to the plan, from defining access and controls to establishing a system for backups.
- Ensure that all staff sign appropriate confidentiality forms, all keys to file cabinets are accounted for, and staff abide by your password policy.
- Communicate your data security plan to staff, highlighting actions they may need to take to protect client data. Also, explain to staff what they should communicate to clients about why their information is being collected and how data security risks are being minimized.
- Educate users about data security risks such as phishing. Conduct simulated phishing attacks if possible.
- Promote awareness by regularly training employees and volunteers in data security procedures.
Review
- Review your data security plan periodically to make sure it still aligns with your current data collection and reporting practices.
- Stay current with antivirus software updates.
- Keep security patches of servers and laptops up to date.