### Improving Datacore: Documentation, Testing, and Low-Code Accessibility #### Context The conversation revolves around improving the **Datacore** plugin through documentation, testing, and making it more accessible to users with limited JavaScript knowledge. The participants discuss their willingness to contribute to the project in areas such as testing and documentation while expressing concerns about the learning curve that comes with using Datacore due to its JavaScript-heavy nature. --- ### Key Points #### 1. **Documentation and Beta Releases** - **Automation for Beta Releases**: Setting up automation for cutting beta releases, likely using the **BRAT** (Beta Release Authoring Tool) plugin in Obsidian, is a priority. This allows users to install and test beta versions easily. - **Documentation**: There’s an active effort to create clear and comprehensive documentation. With a more potent JavaScript engine compared to Dataview, the complexity of Datacore requires thorough guides to help users understand how to use it effectively. - **Example Vault**: The creator plans to set up an **example vault** to help users learn by example. This would include working Datacore queries, YAML structures, and various usage scenarios. #### 2. **The Challenge of a Learning Curve** - **JavaScript Knowledge Barrier**: The current version of Datacore is heavily reliant on JavaScript, making it harder for non-coders or those with limited JavaScript experience to use effectively. - **Low-Code Approach**: A key suggestion is to develop a **low-code interface**. This would allow users to create tables, lists, and other features without needing deep knowledge of JavaScript or the inner workings of the plugin. - Potential solutions include: - A **WYSIWYG** (What You See Is What You Get) editor. - A **builder UI** that guides users through query creation. - **Focus on Core Functionality First**: While low-code features would be highly valuable, the current priority is on core JavaScript features, as this allows more functionality with less UI work. #### 3. **Testing and Feedback** - **Testing**: Contributors without JavaScript expertise can assist by testing various features and providing feedback. Testing will help identify bugs and issues that need fixing, especially since Datacore is in active development. - **Usage Feedback**: Since Datacore is significantly more powerful than Dataview, feedback on **how users interact** with the plugin will help refine the user experience. --- ### Actionable Next Steps #### For Contributors: 1. **Test Datacore Features**: - Set up the **BRAT plugin** to install beta releases of Datacore. - Use Datacore in real-world scenarios (task management, data querying) and provide feedback on the usability of queries, table creation, and performance. - Report bugs and share experiences on what works and what doesn’t. 2. **Contribute to Documentation**: - Help build the documentation by writing guides or tutorials on: - Basic usage (simple queries, setting up Datacore). - Advanced usage (complex queries, integration with YAML). - FAQs and troubleshooting common issues. - Consider contributing to the **example vault**, providing queries and scenarios that can be referenced in the documentation. #### For Plugin Developers: 1. **Low-Code Features**: - Begin ideation for **low-code interfaces** that simplify query creation: - Start with simpler components like a query builder UI that allows users to select fields, filters, and groupings without writing code. - Consider building a **WYSIWYG editor** to create tables and lists without JavaScript knowledge. - Research existing low-code solutions in other plugins for inspiration. 2. **Continuous Integration of User Feedback**: - Gather feedback from early testers to understand pain points in the current Datacore setup. - Use this feedback to prioritize features or changes in future beta releases. - Focus on making documentation as clear and detailed as possible to lower the learning curve. --- ### Conclusion Datacore is an incredibly powerful tool, but its JavaScript-heavy nature presents challenges for less technical users. As the plugin grows, key priorities are building clear and comprehensive documentation, setting up beta releases for easier testing, and eventually implementing low-code interfaces to reduce the learning curve. Contributors can play a major role in testing, documenting, and refining the user experience. ### Expanding Datacore's Query Language: Powerful Filtering with Optimized Syntax This conversation is about the **Datacore Query Language (DQL)** and its evolution, particularly its flexibility, efficiency, and how it can be extended or utilized effectively. Here's a breakdown of the key points and concepts discussed: --- ### Key Concepts 1. **Query Optimization in Datacore**: - Datacore’s query engine has been designed to push as much logic as possible into the query itself, minimizing the need for post-processing (e.g., `.filter()` or `.map()`). - Queries like `datacore.tryFullQuery('rating > 7')` execute efficiently within Datacore’s architecture. This results in major performance optimizations compared to having multiple operations that require filtering or mapping after the initial query. 2. **Syntax Evolution**: - The new Datacore syntax uses `-prefixed fields (e.g., `$title`, `$ctime`) for intrinsic data fields instead of the `file.<field>` format from Dataview. - It fully supports the old DQL syntax but has more advanced capabilities and optimizations. - Example query: `datacore.tryFullQuery('LastName="Johnson"')`. 3. **Handling Complex Queries**: - The query language supports advanced features such as filtering on frontmatter directly in the query (e.g., `datacore.tryFullQuery('@page and rating > 7')`). - Post-query filtering with `.filter` or `.map` is minimized because Datacore encourages using the query itself for most of the filtering. - Fields with spaces need to be handled carefully. For example, to access a field with spaces, you need to use `row["field with space in it"] = value`. This can be improved with shorthand options in the future. --- ### How Query Syntax Works in Datacore The query language in Datacore is defined in the - [ ] **parser** file ([`parser.ts`](https://github.com/blacksmithgu/datacore/blob/8e62a314df259169a28e79c88c2b96172b4151e3/src/expression/parser.ts)) and allows for a wide range of expressions and query structures. The query language is built using **parsimmon**—a powerful parser combinator library that enables complex expression parsing. - **Primitive Parsing**: Datacore supports primitive types like numbers, strings, booleans, and dates. These are parsed using well-defined regular expressions or patterns. - Example: Numbers are parsed using the regex `/^-?[0-9]+(\.[0-9]+)?/`. - **Expressions**: The query parser supports binary operators (e.g., `+`, `-`, `>`, `<`), boolean operations (`AND`, `OR`), and variable identifiers. This allows for creating rich query conditions directly in the query string. - Example: `@page AND rating > 7`. - **Postfix Parsing**: Datacore supports indexing and function calls in the query language, allowing users to access nested data or perform operations like function calls on variables. - Example: Indexing into a list or object: `myList[0]`. --- ### Parsing and Execution in Datacore The Datacore query engine provides efficient ways to parse and execute queries, allowing for fast lookups and filtering. Here’s an outline of the parser components: 1. **Primitive Parsing**: - **Numbers**, **strings**, **booleans**, **tags**, and **dates** are parsed as the foundational elements of any query. - **Dates** are handled with special shorthands like `now`, `today`, `yesterday`, etc., allowing for time-based queries. 2. **Binary Operations**: - Binary operations (`+`, `-`, `>`, `<`, etc.) are parsed to create expressions that can be evaluated. - Boolean combinations (`AND`, `OR`) allow for combining multiple conditions in a single query. 3. **Query Parsing**: - The query language allows for matching tags, checking paths, linking to other pages, and accessing frontmatter or inline fields. - Example queries: - **Tag Matching**: `#myTag` - **File Path Matching**: `path("folder/filename")` - **Frontmatter Access**: `rating > 7` --- ### Examples of Queries in Datacore 1. **Basic Query**: Find all pages with a specific path. ```js datacore.tryFullQuery('@page and path("folder/myfile")'); ``` 2. **Frontmatter Query**: Find pages where the frontmatter `rating` field is greater than 7. ```js datacore.tryFullQuery('rating > 7'); ``` 3. **Combining Queries**: Find pages tagged with `#project` and where the `status` is `active`. ```js datacore.tryFullQuery('@page and #project and status="active"'); ``` 4. **Handling Fields with Spaces**: Access fields with spaces using `row["field name"]`. ```js datacore.tryFullQuery('row["First Name"] = "John"'); ``` --- ### Moving Forward: Simplifying Query Syntax As the query language evolves, simplifying certain aspects like handling fields with spaces is a priority. There could be a shorthand or more user-friendly way of writing such queries in the future. For now, the parser supports a rich set of query conditions, but there is room for improvement in terms of syntax sugar and making the language more intuitive for users, especially those less familiar with JavaScript or complex queries. --- ### Conclusion The **Datacore Query Language (DQL)** is a powerful and optimized way to query, filter, and access data within Obsidian using Datacore. Its parser supports a wide range of expressions and conditions, allowing for complex queries to be executed efficiently. By pushing more logic into the query itself, Datacore minimizes the need for post-query filtering or transformations, resulting in faster execution. If you are new to Datacore, it’s essential to understand the core query syntax and leverage the intrinsic fields like `$title` and `$ctime`. The ongoing evolution of the query language will make it even more accessible and powerful, ensuring that both technical and non-technical users can benefit from its features. ### Datacore: Enhancements, Bug Fixes, and Future Features #### Context This conversation revolves around recent enhancements and bug fixes in the **Datacore** plugin, particularly: - Query language improvements and syntax adjustments. - New features like pagination behavior (scroll-to-top), reusable components (via `dc.require()`), and callout blocks. - Bugs related to tagged blocks, newlines, and formatting issues. --- ### Key Points and Updates #### 1. **Query Syntax: "linkedto" vs "linksto"** - **Issue**: There was confusion between the documented use of `linkedto` and the actual working query syntax `linksto`. - **Resolution**: The correct syntax is `linksto`, as it works in practice, and the documentation should be updated to reflect this. #### 2. **Pagination Behavior: Scroll-to-Top** - **Issue**: When using pagination in long queries, the page would remain scrolled down to the bottom after switching pages. - **Feature**: A new setting, `Scroll on Page Change`, was introduced to fix this. Now, users can set `scrollOnPaging={true}` to ensure that the new page loads at the top when navigating through paginated data. - **Implementation**: This feature is available globally through settings or per table by passing the option directly. #### 3. **Custom Views and Reusable JSX Components** - **Current System**: In Dataview, `dv.view()` allowed the reuse of custom views. In Datacore, this is being replaced by `dc.require()`, which allows users to load and reuse JSX components or JavaScript code from separate files. - **Future Plans**: This feature is in the works and will allow users to break down large queries, CSS styling, and logic into modular pieces, improving maintainability and readability. - **Example Usage (once available)**: ```jsx const { MyCustomComponent } = await dc.require("path/to/customComponent.js"); return <MyCustomComponent />; ``` #### 4. **Callouts** - **New Feature**: Callout blocks are now supported using `<dc.Callout>`. These allow users to create highlighted sections in their notes or tables. - **Example**: ```jsx return ( <dc.Callout title="Important Note"> This is some important content! </dc.Callout> ); ``` #### 5. **Bugs with Tagged Blocks and Newlines** - **Reported Bugs**: - When a block starts at the beginning of the file, the first letter is missing. - Multiple newlines (`\n`) at the start of a file cause: - The first tagged block to be replaced with the next untagged block if there’s exactly one newline between them. - A weird dot appears, similar to bullet list formatting. - **Acknowledgment**: These bugs have been noted, and the team will look into fixing them as part of the plugin's ongoing development. --- ### Actionable Features and Enhancements #### 1. **Scroll-to-Top for Pagination** - If your tables contain multiple pages and you want to ensure that pagination loads the top of the new page: - **Enable in Settings**: Turn on `Scroll on Page Change` in Datacore settings. - **Or use in Tables**: Add `scrollOnPaging={true}` directly to table components: ```jsx <dc.VanillaTable columns={columns} rows={rows} paging={15} scrollOnPaging={true} /> ``` #### 2. **Reusable Components with `dc.require()`** - Once fully implemented, `dc.require()` will allow users to modularize code and reuse components across different queries and files. - **Example**: ```jsx const { TaskTable } = await dc.require("path/to/taskTableComponent.js"); return <TaskTable data={taskData} />; ``` #### 3. **Handling Callouts** - Callouts can be used to create visually distinct sections in notes: ```jsx return ( <dc.Callout title="Warning"> Please note that this task is critical. </dc.Callout> ); ``` #### 4. **Bug Fixes** - Keep an eye out for updates on the tagged block and newline issues. Reporting any further bugs in these areas will help the team prioritize fixes. --- ### Conclusion Recent updates to Datacore have added useful features such as scroll-to-top for pagination, callout blocks, and an upcoming system for reusing components with `dc.require()`. While bugs related to tagged blocks and newlines have been reported, the development team is aware and addressing them. For now, users can explore the improved query syntax, pagination options, and new features while contributing to testing and documentation as the plugin continues to evolve. ### Datacore: Bug Fixes, Pagination Enhancements, and Scroll-to-Top Functionality #### Context In this conversation, several key points and issues regarding Datacore’s functionality are discussed, including: 1. **Edge case bugs with tagged blocks** that have been fixed in version `0.1.10`. 2. **Pagination behavior improvements** such as `ScrollToTop` functionality and pagination buttons positioning. 3. **Filtering improvements and performance fixes** in version `0.1.16`. --- ### 1. **Edge Case Bugs with Tagged Blocks** There was an issue where blocks with certain tags, especially those involving newlines (`\n`) at the start of a file, did not behave as expected. Specifically: - **Tagged blocks** starting with newlines caused parsing issues. - **Blocks without tags** sometimes interfered with those that had tags, leading to unexpected behavior. #### Fix in Version 0.1.10: The off-by-one errors responsible for these issues were addressed, and version `0.1.10` should now handle both cases properly: - Blocks that start with newlines. - Tagged blocks scattered through the document. **Example Code**: ```jsx return function View() { const blocks = dc.useQuery(`@block and #tagged`); return <div> {blocks.map(block => ( <dc.SpanEmbed path={block.$file} start={block.$position.start} end={block.$position.end} /> ))} </div>; } ``` This code fetches and displays blocks tagged with `#tagged` and embeds their content within the view. The issue of blocks not appearing correctly due to newlines or tags has been resolved. --- ### 2. **Scroll-to-Top Functionality in Pagination** #### Issue: Users have reported that **Scroll-to-Top** functionality works for tables but not for other types of list views, particularly when using card-based views or custom pagination components. Despite enabling the `Scroll on Page Change` setting, the content wasn't scrolling to the top when navigating through pages. #### Solution: If you're using a custom list or card view (e.g., a detailed media list) that isn’t part of Datacore’s default table, you'll need to implement your own `scrollToTop` function, as shown in the provided example. This can be done by manually triggering the scroll behavior after the page changes. **Example Scroll-to-Top Function**: ```jsx const scrollToTop = () => { window.scrollTo({ top: 0, behavior: 'smooth' // Adds a smooth scrolling effect }); }; // Example usage within pagination handling const handlePageChange = (page) => { setCurrentPage(page); // Update the current page scrollToTop(); // Scroll to the top after the page changes }; // Example pagination component <TopRightPagination currentPage={currentPage} totalPages={totalPages} setCurrentPage={(page) => { setCurrentPage(page); // Update the page scrollToTop(); // Ensure the page scrolls to the top }} /> ``` This method ensures that the page scrolls to the top whenever the user navigates to a new page, even in custom list or card views where Datacore’s built-in `Scroll on Page Change` setting doesn't apply. --- ### 3. **Filtering on `$file` and Performance Enhancements** #### Issue: In earlier versions, filtering on `$file` (a common way to query files by name) wasn't working as expected. This was a crucial feature for many Datacore users who rely on querying files based on their metadata. #### Fix in Version 0.1.16: The bug preventing `$file` from being used in queries has been fixed in version `0.1.16`, along with performance improvements. Filtering on `$file` should now work properly and more efficiently. **Example Query**: ```jsx datacore.tryFullQuery('@page and $file="filename.md"'); ``` This query filters for pages where the file name matches `"filename.md"`, which is now properly supported and faster. --- ### 4. **Pagination Button Placement** #### Request: A suggestion was made to add pagination buttons at the **top** of tables (or allow them to be placed both at the top and bottom). This would make navigation more convenient when users are dealing with large datasets (100+ items), so they don't have to scroll down to access pagination controls. #### Planned Feature: The Datacore team has agreed that this is a useful feature and plans to add configuration options to control pagination button placement. Users will be able to choose: - **Top** pagination controls. - **Bottom** pagination controls. - **Both** (pagination controls at both the top and bottom of the table). This feature will likely be implemented as a configuration setting within the table component. --- ### Summary of Updates and Enhancements 1. **Tagged Block Bugs**: Issues with blocks starting with `\n` and tagged blocks have been fixed in `0.1.10`. 2. **Scroll-to-Top for Pagination**: - Works for tables by enabling `Scroll on Page Change` or using `scrollOnPaging={true}`. - For custom list views, manually implement `scrollToTop()` in the page change handler. 3. **Filtering on `$file`**: Fully functional and more performant in `0.1.16`. 4. **Pagination Button Placement**: Upcoming feature to allow pagination controls at the top, bottom, or both. --- ### Next Steps for Users - **Upgrade to Version 0.1.16**: Ensure you have the latest version to benefit from bug fixes and performance improvements. - **Manually Implement Scroll-to-Top**: If you're using custom views, implement the `scrollToTop()` function in your pagination logic. - **Watch for Upcoming Features**: Keep an eye on future updates for features like flexible pagination button placement and enhancements to the query language. ### Datacore: Scroll-to-Top for Custom Views, API Exposure, and Ongoing Improvements #### 1. **Scroll-to-Top for Custom Views** **Issue**: Users creating custom card-based list views with pagination have found that the `Scroll-to-Top` functionality isn't working out-of-the-box. **Solution**: Scroll-to-top needs to be implemented manually in custom views using `scrollIntoView()` on the container element. Datacore’s built-in `ScrollToTop` functionality is specific to certain table views but can be adapted for custom card or list views. **Example Implementation**: ```jsx const scrollToTop = () => { document.querySelector('#rootContainer').scrollIntoView({ behavior: 'smooth', // Smooth scrolling effect block: 'start', inline: 'nearest' }); }; // Example usage inside a pagination change handler const handlePageChange = (page) => { setCurrentPage(page); // Update current page scrollToTop(); // Scroll to the top of the view }; // Using pagination controls <TopPagination currentPage={currentPage} totalPages={totalPages} onPageChange={(page) => { handlePageChange(page); }} /> ``` - You can modify this by referencing your container element (`#rootContainer` in this case) and calling `scrollIntoView()` to achieve the smooth scroll behavior when the page changes. - This is similar to the internal Datacore logic for scrolling within tables, but it's essential to manually implement it in your custom views. --- #### 2. **Exposing the Datacore API** **Question**: Users are asking when the Datacore API will be fully exposed for broader use, enabling more custom capabilities for building advanced views and features. **Answer**: The **Datacore API** is already accessible through two primary methods: 1. **`dc` inside Datacore views**: This provides access to core Datacore functionality within JSX or JavaScript views. 2. **`window.datacore`**: This is available globally, providing a broader API that can be used anywhere in Obsidian to interact with Datacore. If you want to leverage Datacore functionality outside of JSX views or in other plugins, you can use `window.datacore` for tasks like querying data, accessing metadata, and interacting with the vault programmatically. **API Documentation**: There is still ongoing work to refine the documentation and provide more examples. A note could be added to the `docs/datastore.md` indicating that `datacorejs`, `jsx`, `ts`, and `tsx` queries are currently available. --- #### 3. **Recent Feature Highlights** 1. **Pagination Enhancements**: - Users have implemented pagination controls at both the top and bottom of list views, making it easier to navigate through large datasets without scrolling all the way down. 2. **Sorting and Filtering**: - Advanced sorting (e.g., by date, title) and filtering (e.g., by media type) are now being applied in custom queries, allowing users to manage complex datasets with ease. 3. **Interactive Elements**: - Features such as clickable media items, favorites management via local storage, and dynamic content resizing based on window size provide a robust and responsive experience. 4. **Media Playback**: - Lightbox integration for YouTube and Vimeo videos allows users to play media directly from the list view. --- #### 4. **Community Feedback and Ongoing Improvements** - **Performance Comparisons**: Users who have converted their workflows from Dataview to Datacore have reported substantial performance improvements, particularly when working with large datasets. - **Nitpick on Empty Values**: It was suggested that the default string for empty values in settings should use `\-` instead of `-` to avoid floating empty lists. This is a minor formatting improvement that can enhance visual consistency in views. - **Future API Improvements**: As Datacore continues to evolve, users are looking forward to expanded API capabilities to emulate database-like functionality, similar to what tools like Notion offer. This will make Datacore even more powerful for work-related use cases and data management within Obsidian. --- ### Conclusion - **Scroll-to-Top for Custom Views**: This can be implemented using `scrollIntoView()` in custom list or card views. The functionality works well when integrated into pagination handlers. - **API Access**: The Datacore API is already available through `dc` in views and `window.datacore` globally. Documentation improvements and more usage examples are on the way. - **Ongoing Enhancements**: Datacore continues to evolve with better pagination, sorting, filtering, and media management capabilities, making it a powerful tool for complex queries and responsive designs in Obsidian. By providing both manual and API-driven solutions, Datacore gives users the flexibility to create highly interactive, performant views for managing tasks, media, and more.