toad.social

1 post1 participant0 posts today

Harald SackBuilding on the 90s, statistical n-gram language models, trained on vast text collections, became the backbone of NLP research. They fueled advancements in nearly all NLP techniques of the era, laying the groundwork for today's AI. F. Jelinek (1997), Statistical Methods for Speech Recognition, MIT Press, Cambridge, MA<a href="https://sigmoid.social/tags/NLP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NLP</a> <a href="https://sigmoid.social/tags/LanguageModels" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LanguageModels</a> <a href="https://sigmoid.social/tags/HistoryOfAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HistoryOfAI</a> <a href="https://sigmoid.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://sigmoid.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://sigmoid.social/tags/historyofscience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#historyofscience</a> <a href="https://sigmoid.social/tags/ISE2025" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ISE2025</a> <a href="https://sigmoid.social/@fizise" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@fizise</a> <a href="https://wisskomm.social/@fiz_karlsruhe" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@fiz_karlsruhe</a> <a href="https://fedihum.org/@tabea" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@tabea</a> <a href="https://sigmoid.social/@enorouzi" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@enorouzi</a> <a href="https://fedihum.org/@sourisnumerique" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@sourisnumerique</a>

Holle Meding🔠 Panel: More than Chatbots: Multimodal Large Language Models in Humanities WorkflowsAt <a href="https://mastodon.social/tags/DHd2025" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DHd2025</a>, Nina Rastinger explores how well <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> handles abbreviations & NER:✅ NER works well, even with small, low-cost models ❌ Abbreviations are tricky—costs & resource demands skyrocket 🚀 GPT o1 improves performance, even on abbreviations, but remains resource-intensive Balancing accuracy & efficiency in text processing remains a challenge! ⚖️<a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/NER" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NER</a> <a href="https://mastodon.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.social/tags/DigitalHumanities" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DigitalHumanities</a>

LavX NewsIntroducing TRRE: The Next Evolution in Text Processing with Transductive Regular ExpressionsTransductive Regular Expressions (TRRE) offer a groundbreaking approach to text editing and pattern matching, extending traditional regex capabilities. With a command line tool that redefines how deve...<a href="https://news.lavx.hu/article/introducing-trre-the-next-evolution-in-text-processing-with-transductive-regular-expressions" rel="nofollow noopener noreferrer" target="_blank">https://news.lavx.hu/article/introducing-trre-the-next-evolution-in-text-processing-with-transductive-regular-expressions</a><a href="https://mastodon.cloud/tags/news" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#news</a> <a href="https://mastodon.cloud/tags/tech" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tech</a> <a href="https://mastodon.cloud/tags/TransductiveRegex" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TransductiveRegex</a> <a href="https://mastodon.cloud/tags/FiniteStateTransducer" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FiniteStateTransducer</a> <a href="https://mastodon.cloud/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a>

unixbhaskarUmmmm<a href="https://mastodon.social/tags/linuxadmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#linuxadmin</a> <a href="https://mastodon.social/tags/emacs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#emacs</a> <a href="https://mastodon.social/tags/tool" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tool</a> <a href="https://mastodon.social/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a> <a href="https://mastodon.social/tags/lisp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#lisp</a> <a href="https://mastodon.social/tags/textprocessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#textprocessing</a> <a href="https://sites.google.com/site/steveyegge2/the-emacs-problem" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://sites.google.com/site/steveyegge2/the-emacs-problem</a>

Steven P. Sanderson II, MPH🐧 Struggling with text formatting in Linux? You're not alone! My new article breaks down the essentials for beginners.Read more at <a href="https://www.spsanderson.com/steveondata/posts/2025-01-24/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.spsanderson.com/steveondata/posts/2025-01-24/</a> and let’s discuss!<a href="https://mstdn.social/tags/Programming" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Programming</a> <a href="https://mstdn.social/tags/linux" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#linux</a> <a href="https://mstdn.social/tags/tech" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tech</a> <a href="https://mstdn.social/tags/printf" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#printf</a> <a href="https://mstdn.social/tags/nl" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nl</a> <a href="https://mstdn.social/tags/fold" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fold</a> <a href="https://mstdn.social/tags/groff" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#groff</a> <a href="https://mstdn.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mstdn.social/tags/Formatting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Formatting</a> <a href="https://mstdn.social/tags/Blog" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Blog</a> <a href="https://mstdn.social/tags/CLI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CLI</a> <a href="https://mstdn.social/tags/CommandLineTips" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CommandLineTips</a>

Steven Sanderson💡 Struggling with text processing in Linux?My latest blog post at <a href="https://www.spsanderson.com/steveondata/posts/2025-01-17/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.spsanderson.com/steveondata/posts/2025-01-17/</a> breaks down common challenges and offers practical solutions! Learn how to tackle duplicates and sort data with ease. 📊🔍<a href="https://rstats.me/tags/Text" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Text</a> <a href="https://rstats.me/tags/Blog" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Blog</a> <a href="https://rstats.me/tags/Technology" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Technology</a> <a href="https://rstats.me/tags/Textprocessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Textprocessing</a> <a href="https://rstats.me/tags/Programming" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Programming</a> <a href="https://rstats.me/tags/Linux" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Linux</a> <a href="https://rstats.me/tags/Help" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Help</a> Let me know what you think!

Steven P. Sanderson II, MPH💡 Struggling with text processing in Linux?My latest blog post at <a href="https://www.spsanderson.com/steveondata/posts/2025-01-17/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.spsanderson.com/steveondata/posts/2025-01-17/</a> breaks down common challenges and offers practical solutions! Learn how to tackle duplicates and sort data with ease. 📊🔍<a href="https://mstdn.social/tags/Text" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Text</a> <a href="https://mstdn.social/tags/Blog" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Blog</a> <a href="https://mstdn.social/tags/Technology" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Technology</a> <a href="https://mstdn.social/tags/Textprocessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Textprocessing</a> <a href="https://mstdn.social/tags/Programming" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Programming</a> <a href="https://mstdn.social/tags/Linux" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Linux</a> <a href="https://mstdn.social/tags/Help" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Help</a> Let me know what you think!

Veronica Olsen 🏳️‍🌈🇳🇴🌻Back when I first wrote text processing code in the 90s on my Amiga 1200, I always used the ¤ symbol as a placeholder character for splitting and replacing to exclude things I wanted skipped without affecting character count. It was available on the Norwegian keyboard, and practically never used in text.Recently I discovered that Unicode has two "Not a character" symbols perfect for the same usage: \uFFFE and \uFFFF.They can be really useful!<a href="https://mastodon.online/tags/Code" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Code</a> <a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Unicode</a>

Veronica Olsen 🏳️‍🌈🇳🇴🌻2. Immediately after the split, replace U+FFFF with newline, but keep both versions of the line, and pass the one with the U+FFFF to the text paragraph parser. Everything else (like headings) gets the cleaned one.3. After paragraph lines with a single break between them (belonging to the same paragraph) have been processed, THEN I replace the U+FFFF characters there.It seems to work, but it took me like 3-4 hours to crack. 😅 4/4<a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Unicode</a>

Veronica Olsen 🏳️‍🌈🇳🇴🌻I tried using the alternative line and paragraph separators from Unicode, but splitlines accepts them too. Then I discovered these Unicode characters:U+FFFE <noncharacter-FFFE> not a character. U+FFFF <noncharacter-FFFF> not a character.The solution, then was:1. Replace all occurrences of [br] with or without a trailing newline, using regex pattern "(?i)(?<!\\)(\[br\]\n?)", with a U+FFFF character.3/4<a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Unicode</a>

Veronica Olsen 🏳️‍🌈🇳🇴🌻This works fine in principle, but it is incredibly hard to figure out exactly when to make the replacement.For instance, if I do it too early, the parser will split on the breaks as I use splitlines() early on. If I do it too late, I get double line breaks some places.2/4<a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Unicode</a>

Veronica Olsen 🏳️‍🌈🇳🇴🌻I've been struggling with solving an issue with my text editor project. The editor is plain text and uses a blank line to separate paragraphs.The editor has an option to preserve or not preserve single line breaks inside paragraphs when generating the output.However, some users want to not preserve them, but still want to be able to add hard breaks sometimes. So I've been trying out using [br] as a hard break shortcode.1/4<a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.online/tags/Unicode" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Unicode</a>

Veronica Olsen 🏳️‍🌈🇳🇴🌻I started working on a Python class to write MS Office Word documents from already tokenized formatted text. It took me 5 hours to get a working version that can handle most of the formatting I need.I have already done this with the Open Document format. It took me significantly longer, but I do steal some code from that code for DocX.That said, DocX is actually easier to generate the XML for it turns out.<a href="https://mastodon.online/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.online/tags/Code" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Code</a> <a href="https://mastodon.online/tags/Documents" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Documents</a> <a href="https://mastodon.online/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a>

ZacOnce again I am shocked by the speed at which some of the scripts that I create run on my Mac. I had a Shortcut that read the text from a 100 page PDF and then saved all the data into individual files.I had a Node.js script that took another file and parsed it to take the data from each marked chapter and dump that into a markdown file.Both took seconds to complete. So fast that I thought they didn't work.<a href="https://home.social/tags/MacOS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MacOS</a> <a href="https://home.social/tags/Shortcuts" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Shortcuts</a> <a href="https://home.social/tags/Node" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Node</a> <a href="https://home.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://home.social/tags/Speed" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Speed</a>!!!!!

Jim Donegan 🎵 ✅<a href="https://mastodon.scot/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatGPT</a> with <a href="https://mastodon.scot/tags/RobMiles" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobMiles</a> - <a href="https://mastodon.scot/tags/Computerphile" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Computerphile</a> <a href="https://www.youtube.com/watch?v=viJt_DXTfwA&ab_channel=Computerphile" rel="nofollow noopener noreferrer" target="_blank">https://www.youtube.com/watch?v=viJt_DXTfwA&ab_channel=Computerphile</a> <a href="https://mastodon.scot/tags/ComputerScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ComputerScience</a> <a href="https://mastodon.scot/tags/LanguageModel" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LanguageModel</a> <a href="https://mastodon.scot/tags/LanguageModels" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LanguageModels</a> <a href="https://mastodon.scot/tags/LanguageProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LanguageProcessing</a> <a href="https://mastodon.scot/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a> <a href="https://mastodon.scot/tags/ComputerLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ComputerLearning</a> <a href="https://mastodon.scot/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.scot/tags/ArtificialIntelligence" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ArtificialIntelligence</a> <a href="https://mastodon.scot/tags/GPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GPT</a> <a href="https://mastodon.scot/tags/GPT3" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GPT3</a> <a href="https://mastodon.scot/tags/AILearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AILearning</a> <a href="https://mastodon.scot/tags/AITraining" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AITraining</a> <a href="https://mastodon.scot/tags/OpenAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenAI</a>

clar fonI decided to finally clean up my templating language design based upon feedback from <a href="https://social.emisocks.com/@socks" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@socks</a> and actually commit the result to a repo which will eventually, at some distant point in the future, contain a Rust implementation.in case you're curious, it's my attempt to make a minimal "mustache but not bad" templating language that I can use for various projects<a href="https://vc.ltdk.xyz/ltdk/wright" rel="nofollow noopener noreferrer" target="_blank">https://vc.ltdk.xyz/ltdk/wright</a><a href="https://toot.cat/tags/RustLang" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RustLang</a> <a href="https://toot.cat/tags/Programming" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Programming</a> <a href="https://toot.cat/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a>

g z thompsonokay today's "put together presentation of ideas for tomorrow's meeting" has determined that i really need to learn a lot more about NLPwhich i've been saying for months anyway but today it's like THIS IS REALLY NECESSARY NOWSo: what are essential NLP resources for a statistician/data scientist/ML practitioner? <a href="https://scholar.social/tags/NLP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NLP</a> <a href="https://scholar.social/tags/ML" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ML</a> <a href="https://scholar.social/tags/DataScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScience</a> <a href="https://scholar.social/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://scholar.social/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> <a href="https://scholar.social/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://scholar.social/tags/chatbot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#chatbot</a> <a href="https://scholar.social/tags/TextProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextProcessing</a>

Recent searches

Search options

Administered by:

Server stats:

#textprocessing