vim-scripts news feed

contribute to vim-scripts.org

2011-10-24T00:00:00-07:00

Hello Vimmers!

All of you probably know that vim-scripts.org mirrors all vim.org scripts on Github making them available as git repos.

Scott, the author of vim-scripts.org did a great job with implementing and running it (at his own expense btw). Still it can be even better!

For obvious reasons Scott's open-source-hours are limited as well as mine. Therefore in order to improve vim-scripts.org we're looking for those of you interested in contributing your expertise.

Here's general outline on what needs to be done:

new website theme, logo, favicon
Jekyll based template fixes: pagination, news dupes
language improvements all over the site: edit down
all web-site UX/UI issues
many scraper issues

Yeah, lots to be done!

In case you cannot contribute with your skills you can always donate to support the project.

Discussion is welcomed so make sure to leave your comment.

Thanks all, let's make Vim ecosystem better!

References:

website contributor how-to

Picky About Timezones

2011-06-23T00:00:00-07:00

Github has started refusing commits created by the scraper. Due to a missing zero in the committer date, there's a chance that all script repositories will have to be regenerated.

We live in interesting times.

The good news is that the scraper is back online and caught up so, other than that, things should be back to normal.

The Story

Up until last week, GitHub was happy to accept commits with a time zone of -700. Now, presumably due to a Git upgrade, they only accept pushes with four digit time zones: -0700. The others are refused.

Unfortunately, every vim-scripts commit has been generated with a time zone of -700.

How did this happen?

Back in the beginning of time, the scraper would shell out to the git executable any time it needed to manipulate a repository. Picture a sea of calls like system("git tag -a -m ...") and hoping that everything went OK. It was slow, ugly, unreliable, and confusing.

Then I tried switching to Grit. This made things prettier because I could use Grit objects instead of passing text strings everywhere, the errors became more understandable, and there were a few other benefits. However, there were some serious drawbacks too: it was continually changing the current directory, confusing itself, and it still shelled out for everything. It was pretty clear that Grit was written with GitHub's needs in mind and, if I wanted to use it in the scraper, I would have to commit to some serious work.

Writing Git bindings before even starting to work on the scraper would have taken too much time. Nowadays libgit2 could have solved this but, at the time, this looked like a death blow.

Enter Gitrb

The vim-scripts project has been amazingly lucky a few times in its history. Just when everything looked bleak and it was time to pack it in, along would come a person or a gem that put things back on track. In this case it was Daniel Mendler's gitrb gem.

Gitrb is a real, object-based API in 100% Ruby that directly manipulates the on-disk Git repository. It's fast, it's easy to use, it's even kind of fun. The few issues that I found were easy to solve and Daniel was very responsive to modifications. The scraper was rolling again.

One of the issues that I ran into was that Gitrb produced invalid time zones in the western hemisphere: --700, which git would immediately refuse. It's easy to understand, Daniel lives to the east of the Meridian. No problem, a quick patch, and git is happy with time zones of -700.

The Bug Comes Home

Fast-forward nine months. Nine wonderful months free of git-related issues.

Just as the RSS feed comes back online, GitHub starts refusing every push we make. It took a while to find the problem because git log --pretty=fuller shows the correct -0700 time zone. It's only when you use git log --pretty=raw that the horrifying truth becomes apparent.

Git didn't care before. The check was added in April 2010, a month before I wrote the Gitrb time zone fix.

Fixing gitrb was easy. Fixing all the broken repos, now that's going to be a challenge.

What Does it Mean?

Well, nothing needs to change immediately. Thanks to Bundler, the scraper started using the fixed gitrb right away. GitHub is accepting the fixed commits and we should be caught up in a day (update: 4 hours later, we're caught up).

Of course, we still need a proper fix. The only way that I see is to regenerate every repository and re-upload everything to github. This is not unprecedented. We did it back in October 2010 (start, finish), before beta, when the scraper had become much smarter at parsing dates and email addresses.

Of course, that would mean that all local repos will have to be deleted and re-cloned. That should be fairly painless if you're using Vundle or vim-update-bundles. For people maintaining their plugins by hand, though, this might be an unpleasant surprise.

Luckily, we don't have to do anything immediately. We will take the time to implement the proper fix, whatever it turns out to be.

Archive Armageddon

2011-06-23T00:00:00-07:00

This is a quick description of two scraper stalls that happened in May.

Earlier we discussed scraper stalls and how they prevent bugs from slipping through the cracks. This post talks about two specific stalls from mid-March. If it seems like old news, that's because this post has been sitting half-written since June 3, silently ignored as all sorts of weird and wonderful new ways to stall the scraper have been discovered.

Sensing Files

A script author uploads a text file or an archive to vim.org, the scraper downloads it and stuffs it into a git repository, and Bob's your uncle. No problem!

The trick is, mistakes run wild on vim.org. The scraper needs to try to clean things up. And that quickly turns into guesswork.

The most common mistake script authors make is putting the wrong extensions on the files they upload. A .tgz is not actually gzipped, a .bz2 is actually gzipped, and so on.

We can sniff the file's magic to discover what it really is but that's often wrong too! Besides, Java jarfiles sniff as zipfiles, some archive formats sniff as the files they contain, and so on. While it's more reliable than the file's extension, it turns out we can't rely on file magic either.

The solution to this problem has produced the most complex part of the scraper: the sense_file routine. It tries to reconcile the file's extension and magic to determine the best way to handle it. Grown through tedious trial and error, sense_file has become big and ugly but, implausibly, it almost always works.

The Case of the Misleading Extension

In mid-May, a script author uploaded a .tgz file that was only tar encoded. The scraper didn't anticipate this, couldn't figure out which claim to trust, so it bailed out.

I'm surprised it took this long for this particular mistake to show up! I would have bet money that someone would have done it before 2005.

This was easy to solve once it was clear what was going on. Result: a 15 minute fix and a 2 day stall.

Archive Formats

Most of the complexity may be in the sense_file routine (above), but most of the time spent coding the scraper has gone into figuring out just how to expand all the grotty archive formats on vim.org. They have .zip, .tar, .tar.gz, .tar.bz2, .xz, .7z, .rar, .vba, and so on With 12,000 files written by 1800 authors, there are going to be a lot of incompatible, corrupt, and just plain weird files to figure out.

Zip and vimball are tied for the most heinous of all. We'll talk about zip because it's common, interesting, and caused the longest stall in May. Let's hope that we never have the need to discuss vimball (though you can grep the scraper's commit history for 'vimball' if you're morbidly curious).

Zip a Dip

Way back in history, before vim-scripts was public, the scraper used the Rubyzip gem to unzip files. Unfortunately, Rubyzip has an obscure bug that is unlikely to ever be fixed. This project means well but will never be mature enough to handle the variety of zipfiles that we're processing.

So, throw that code out and try again.

The scraper was converted to shell out to the system's unzip command. This was hugely more reliable. The lesson is clear: whenever possible, use the same tool everyone else is using. Especially when you're dealing with a lot of crazy data, code maturity trumps pretty much everything else.

That said, even unzip needed some workarounds.

On Unixy systems, unzip tends to be a variant of the venerable Infozip command. This code base dates all the way back to the 80s (remember far ptrs?). It's beautifully stable but maintenance appears to be light and sporadic. And the code, oh the code... *shudder* If you want to remember what conditional macro hell was like, browse the Infozip source.

Anyhow, that worked swimmingly for nine months. Then, a week ago, a script author uploaded the first zipfile that Infozip can't handle:

skipping: plugin/checklist.vim    need PK compat. v6.3 (can do v4.6)
skipping: readme.markdown         need PK compat. v6.3 (can do v4.6)
skipping: syntax/chklst.vim       need PK compat. v6.3 (can do v4.6)

Back in 2006, pkware updated the zipfile specification and Infozip still can't handle it. And, given its slow rate of development, 6.3 compatibility is probably years away. Time to find another unzipper!

It turns out Igor Pavlov's 7-zip utility can uncompress zipfiles, handles the new format, and the scraper is already using it to uncompress .7z files anyway! With just a small tweak we can use 7z for all zipfiles.

Unfortunatley, 7z can't reliably unzip DOS archives that use '\' as a path separator. There are currently seven zipfiles in the archives that 7z can't uncompress. This looks easy to fix so I've filed a feature request.

But, once again, that leaves us with a compromise to make. For now, the scraper will continue to use Infozip to unzip regular zipfiles and special case any new zipfiles that Infozip can't handle. (If a v6.3 zipfile that uses '\' as a path separator shows up then we're hosed).

When the newer zip format becomes more widely used and the special-case list is getting too long, we can switch to using 7z by default and then hard-code the seven files that require Infozip. Another approach would be to sniff the zipfile to see what path separator it's using and then use that to determine which unzipper to use.

With a little time and luck, hopefully the feature request will receive some love, 7z will properly handle all zipfiles, and there will be no need to special-case anything anymore.

And that's how a small scraper stall can turn into a big research project!

Vim.org RSS Errors

2011-06-12T00:00:00-07:00

As mentioned in the previous news posting, the scraper uses the RSS feed to discover which scripts have been updated.

Unfortunately, for the past few days, vim.org's RSS Feed has been producing an error page. vim.org is still up, so authors are still writing and uploading scripts, we just can't figure out which ones.

Not unless we download all 3622 scripts one by one. It's possible, and we'll do it, but it takes a lot more time and bandwidth.

This means that, until vim.org's RSS feed returns, updates will still happen but they will be a lot slower!

Scraper Stalls

2011-06-03T00:00:00-07:00

Observant vim-scripts users noticed that recently the scraper was stalled for 3 weeks. The problems that caused this have been fixed and the scraper is rolling again.

While a three week outage is never good, the occasional outage is expected and even a sign of health. If you're interested, here's a bit more about why the scraper stalls.

Whence Stalls

The scraper watches vim.org's RSS feed, processes the updated scripts, and goes back to sleep. If there's a problem, it bails out and refuses to process any more scripts until a human intervenes and fixes the problem.

Why not just skip the offending script and keep on scraping? Surely a coauthor of the Retryable gem knows how to rescue from exceptions! Let the scraper keep churning, ignore the broken script, and it would take months for anyone to notice or care that anything is wrong.

Here's the catch: fixing scraper issues can get pretty tedious. It's not easy to find time to debug code, especially when most of the time it turns out to be nothing more than a .zip extension on an uncompressed .gif.

Programmers have lots of cool things to work on. You need to yell pretty loud if you want to attract their attention.

So, for the sake of argument, let's say the scraper just kept on cruising. Every week or two another script author finds a new way of breaking things and a few more scripts fail. Unless it's a high profile script like rails.vim, no maintainer shows up to mend the problems. After some difficult years, the scraper finds that it can only scrape taglist.vim. No maintainer is willing to devote the man month of time to repair the hundreds of small problems everywhere. Suddenly finding itself friendless and jobless, unforked and unloved, the scraper sits forever idle on GitHub.

The Upshot

The scraper walks a fine line: it needs to ensure its problems are big enough to get fixed and yet small enough to be fixable. That's why it yells loud and early. Hopefully the problem is small, the fix quick, and the entire project will be back on track in a day or two.

Forcing the most reliable behavior in the small picture can sometimes be destabilizing in the big picture. NetFlix's Chaos Monkey is another example. It might be counterintuitive but it certainly seems to work.

And that's why scraper stalls are a normal part of life. They happen occasionally, get crushed like a minor irritation, and things go back to normal.

If you notice a stall, first smile because it's an indication that script authors are innovating. Vim is still thriving. Then, if nothing appears to be happening, help the scraper yell for help.

Postscript

I must say, I'm surprised that vim-scripts works as well as it does. Web scraping is messy business that continually breaks at random and inopportune times. Add to that the complete free-for-all that is vim.org/scripts (rename your script at any time, invent your own way of deleting your scripts, no standard archive or plugin format, etc) and, in theory, you have an inky soup that just won't convert into Pathogened git repositories. Last August I was starting to worry that every morning I would wake up and immediately need to fix all the scrape problems created uring the previous day.

Happily, script authors tend to make the same mistakes and find the same solutions as one other. If I find a workaround for one script, ten unrelated scripts might be also be fixed (five of which haven't even been written yet).

It was rocky going at first, but once the scraper reached a certain maturity, it proved that it could wade through the soup almost completely on its own.

Of course, the scraper will never be complete, and there will always be stalls. But, when I go on a two week vacation, I'll be surprised if things are stalled when I get back. And that's good enough!

Planned Outage Sunday Night

2011-04-08T00:00:00-07:00

vim-scripts.org will be down for 1/2 hour on Sunday night Pacific time. We will move the last non-script repository out of the GitHub vim-scripts account. When finished, the vim-scripts account will contain 100% scripts and its RSS feed will contain 100% script news.

The catch is, of course, that the one remaining repository is the one hosting vim-scripts.org.

Comments Are Live

2011-03-21T00:00:00-07:00

Discuss.

Combining Issue Trackers

2011-03-21T00:00:00-07:00

vim-scripts.org used to have one issue tracker for the website and another issue tracker for the scraping robot. Turns out this was needlessly confusing. The issues for the website have been merged into the scraper and now there is a single issue tracker for the entire project.

The issue tracker is also being moved from the vim-scripts account to vim-scraper (the GitHub support request). If the vim-scraper link 404s, then the move hasn't happened yet. If it resolves then the move is complete.

UPDATE: The move is done. Thanks GitHub!

Unfuddling Email Addresses

2011-02-20T00:00:00-08:00

Some time around February 15, vim.org modified the way it presents the email adddresses of script authors. They just replaced @ and . with images, but this caused the vim-scripts.org scraper to push 29 invalid repositories before it could be worked around.

The following scripts have been fixed and re-uploaded:

activity-log
AuthorInfo
CCTree
Chess-files-.pgn-extension
ColorSelect
csindent.vim
css3
csv.vim
CTAGS-Highlighting
earendel
fecompressor.vim
Flog
funprototypes.vim
gtk-vim-syntax
Headlights
Hoogle
IniParser
iptables
lexctwolc-Syntax-Highlighter
LispXp
Lucius
mxl.vim
powershellCall
SuperTab-continued.
taskpaper.vim
utlx_interwiki.vim
VimCalc
vim-paint
vundle

There are a number of other changes (like this) queued up for vim-scripts.org but $dayjob has been taking up a lot of time. Big thanks to gmarik for keeping keeping the fires burning.

No Apple Files Part 2

2010-11-23T00:00:00-08:00

The previous attempt at eradicating bogus Apple files didn't quite get all of them. In addition to removing .DS_Store and ._DS_Store files, now we remove all ._* files too.

The following scripts were affected:

CodeOverview
PIV
cuteErrorMarker
slurper.vim

With this change, the Scraper now produces the exact same results on both Linux and MacOS.

Scraper Code Moved

2010-11-22T00:00:00-08:00

The scraper and its associated script and data files change with pretty much every scrape. They always bounce to the top of the newsfeed, drowning out the rest of scripts.

To address this, the scraper's code has been moved to the vim-scraper user. The old repo, owned by the vim-scripts user, still exists but only so its issues don't get deleted. Since it's not receiving any more commits, it will no longer be a prominent news non-item.

The jekyll pages are also noisy, and haven't been moved either.

Other than these two repositories, however, vim-scripts is now 100% scripts! The news feed should be a lot quieter.

Also, in unrelated news, :REPL has been renamed to REPL.

1903 - REPL

Suppress Apple DS_Store Files

2010-11-15T00:00:00-08:00

Apple software creates Desktop Services Store files to keep track of folder locations and icons. Since they're not needed in Vim scripts, the scraper removes them.

The plugins that no longer carry .DS_Store files:

1929 - DBGp-client
2424 - Transmit-FTP
2441 - pyflakes.vim
2507 - 0scan
2540 - snipMate
2674 - cocoa.vim
2813 - MailApp
2852 - perlomni.vim
2883 - growlnotify.vim
2890 - slurper.vim

Also, minor fixups to two scripts: put tags on the correct commits, and change 'Syntax' to 'syntax' so the script will work on case-sensitive filesystems too.

0790 - python.vim--Vasiliev
2651 - fitnesse.vim

Trimming Archive Hierarchy Revisited

2010-11-15T00:00:00-08:00

Thanks to an imprecise regex in the hierarchy fix, the scraper didn't repair as many scripts as it could have.

The following scripts have had their hierarchies fixed even more:

0775 - vimcdoc
0860 - inform.vim
1000 - gtk-vim-syntax
1431 - checksyntax-B
1616 - tdl.vim
1770 - etk-vim-syntax
1795 - autoloadTemplate.vim
1929 - DBGp-client
2011 - dbext4rdb
2301 - Gembase-dml-plugins
2484 - textobj-indent
2627 - exVim
2665 - HtmlHelper
2700 - PKGBUILD
2716 - textobj-syntax
2743 - greek_polytonic.vim
2905 - vim-addon-manager
3027 - grsecurity
3081 - JavaScript-Indent
3227 - Simple-Javascript-Indenter

Also, two horribly broken vimballs have been suppressed from NrrwRgn.

3075 - NrrwRgn

Trimming Archive Hierarchy

2010-11-13T00:00:00-08:00

A small refactoring allows the scraper to be a lot smarter about cleaning up hierarchies. What do I mean? Stuff like this:

script-0.1/plugin/script.vim
script-0.1/doc/script.txt

That leading "script-0.1" should be removed. It's not easy to determine which directories can be removed and which ones need to be kept, especially for a small script scraping robot. There are as many different ways of naming directories as there are script authors.

The following scripts have been fixed:

0023 - EnhCommentify.vim
0067 - nqc.vim
0090 - vcscommand.vim
0100 - TagsBase.zip
0213 - c.vim
0222 - mu-template
0280 - netdict
0300 - Vimacs
0336 - lh-cpp-ftplugins
0365 - bash-support.vim
0391 - LaTeX-functions
0403 - eruby.vim
0459 - vim-templates
0468 - VEC
0472 - mup.vim
0556 - perl-support.vim
0560 - MPD-syntax-highlighting
0584 - pyljpost.vim
0663 - VimDebug
0684 - Comment-Tools
0722 - cygwin.vim
0775 - vimcdoc
0835 - systemc_syntax.tar.gz
0860 - inform.vim
0861 - VikiDeplate
0889 - EvalSelection.vim
0906 - Editable-User-Interface-EUI-eui_vim
0922 - svncommand.vim
0996 - tbf-vimfiles
1000 - gtk-vim-syntax
1021 - feralstub.vim
1030 - scalefont
1038 - vimgrep.vim
1039 - array.vim
1053 - browser.vim
1054 - InteractHL.vim
1055 - curBuf.vim
1059 - pygtk_color
1060 - Vive.vim
1095 - TabLaTeX
1127 - vim2ansi
1160 - tSkeleton
1173 - tComment
1177 - OAL-Syntax
1196 - omlet.vim
1232 - DoTagStuff
1284 - TortoiseSVN.vim
1306 - cguess
1311 - vimplate
1431 - checksyntax-B
1442 - XML-Completion
1460 - glark.vim
1582 - vim-addon-background-cmd
1584 - Erlang-plugin-package
1616 - tdl.vim
1620 - scvim
1631 - gnuchangelog
1633 - svncommand-tng
1694 - SnippetsMgr
1724 - pyljvim
1730 - tAssert
1748 - toggle_word.vim
1762 - aspnetcs
1770 - etk-vim-syntax
1771 - woc
1810 - Infobasic-Set-Syntax-FTDetect-FTPlugi
1911 - SuperPre
1915 - tbibtools
1929 - DBGp-client
1939 - autofmt
1967 - gvim-extensions-for-TALpTAL
2011 - dbext4rdb
2040 - tcalc
2098 - fakeclip
2100 - textobj-user
2101 - textobj-datetime
2114 - Torquescript-syntax-highlighting
2193 - fpdf.vim
2213 - VimTAP
2219 - Limp
2245 - upf.vim
2250 - common_vimrc
2263 - xtemplate
2275 - textobj-fold
2276 - textobj-jabraces
2301 - Gembase-dml-plugins
2305 - Textile-for-VIM
2335 - metarw
2336 - metarw-git
2337 - ku
2343 - ku-bundle
2344 - ku-metarw
2348 - Chimp
2355 - textobj-lastpat
2382 - Note-Maker
2393 - QuickTemplate
2410 - ku-args
2415 - textobj-diff
2417 - Merb-and-Datamapper
2425 - arpeggio
2483 - Gorilla
2484 - textobj-indent
2501 - VimClojure
2523 - Jifty-syntax
2610 - textobj-entire
2619 - textobj-function
2620 - neocomplcache
2622 - ku-quickfix
2624 - ProtoDef
2627 - exVim
2657 - VOoM
2665 - HtmlHelper
2692 - operator-user
2700 - PKGBUILD
2715 - UltiSnips
2716 - textobj-syntax
2729 - jslint.vim
2771 - Conque-Shell
2782 - operator-replace
2787 - Melt-Vim
2793 - hybridevel
2805 - assistant
2858 - textobj-fatpack
2877 - spinner.vim
2878 - RDoc
2890 - slurper.vim
2905 - vim-addon-manager
2907 - hlint
2933 - vim-addon-fcsh
2934 - vim-addon-actions
2940 - vim-addon-mw-utils
2957 - load_template
2968 - ikiwiki-nav
3012 - vspec
3017 - Cppcheck-compiler
3020 - EasyOpts
3069 - Slidedown-Syntax
3071 - reimin
3081 - JavaScript-Indent
3122 - PySuite
3124 - vim-addon-sbt
3125 - PIV
3196 - gedim
3207 - bundler
3227 - Simple-Javascript-Indenter
3264 - Vimtodo
3290 - RevealExtends
3293 - numbered
3303 - smartbd
3304 - Gundo

This was the last known big deficiency in the scraper. Hopefully it only needs minor cleanups from here on out.

Big thanks to rygwdn for the motivation to make this invasive a change.

Suppressing VCS Directories

2010-11-13T00:00:00-08:00

The scraper now suppresses common VCS dirs (.git, .hg, .bzr, and .svn). The following scripts were affected:

2890 - slurper.vim.stats
2960 - bisect.stats
3081 - JavaScript-Indent.stats
3224 - VIM-Color-Picker.stats
3310 - gdb4vim.stats

Also, to unify some code paths, the scraper no longer uses the minitar gem to untar files. It now shells out to the unix utility, just like it does for zip, rar, 7z, and xv. This fixed a single script:

1608 - Chrome syntax script

A quick ramble offtopic... I've had zero troubles with Ruby's minitar (as opposed to rubyzip, which has been painful), so it comes as a bit of a surprise to me that it hasn't been doing a perfect job. This just reinforces my suspicion that it's a bad idea to rely on little-used gems to perform a complex task when you can just shell out to a tool that's used and tested every day (probably every minute).

Shell out. It may not be as pure but it's a heck of a lot more reliable.

TeX Script Location

2010-11-09T00:00:00-08:00

Thanks to DataWraith for reporting that script 284's files weren't in the correct location. TeX scripts need to go in ftplugin/tex, but the scraper just put them in ftplugin.

The two scripts affected by this have been regenerated:

0284 - unilatex.vim
1095 - TabLaTeX

Also, it turns out that tex.vim has the wrong script type. The correct type is now hard-coded into the scraper.

0093 - tex.vim

Autoload and After directories

2010-11-07T00:00:00-07:00

A lot of plugins put files in autoload or after directories, which are themselves nested in other directories. This is perfectly valid, of course, but the scraper tended to move them from autoload/plugin to the regular plugin directory.

Fixed now. These are the affected scripts:

0099 - mail.tgz
0222 - mu-template
0336 - lh-cpp-ftplugins
0575 - gnu-c
0614 - CRefVim
0835 - systemc_syntax.tar.gz
1048 - R.vim
1052 - The-Mail-Suite-tms
1232 - DoTagStuff
1291 - luarefvim
1318 - snippetsEmu
1442 - XML-Completion
1520 - OmniCppComplete
1727 - Templeet
2099 - xml_autons
2294 - SwapIt
2385 - sqlite_c
2417 - Merb-and-Datamapper
2495 - Pimp
2534 - d.vim--Suhoverhov
2540 - snipMate
2627 - exVim
2628 - Vim-R-plugin
2674 - cocoa.vim
2740 - vimper
2935 - superSnipMate
2954 - hypergit.vim
3125 - PIV
3154 - vorax
3200 - Haskell-Conceal
3265 - cSyntaxAfter

Fixed a Scraper Bug

2010-11-06T00:00:00-07:00

The scraper had a bug where it would mistakenly think that .vimrc files (or .vimanything) were .vim files and throw them into plugins. This missing dollar sign caused 25 scripts to be generated with files in the wrong place.

0213 - c.vim
0365 - bash-support.vim
0369 - atomcoder-vim
0410 - muf.vim
0517 - TVO--The-Vim-Outliner
0556 - perl-support.vim
0570 - GTK-
0795 - xterm16.vim
0838 - macro.vim
1000 - gtk-vim-syntax
1540 - mousefunc-option-patch
1541 - GetChar-event-patch
1770 - etk-vim-syntax
2126 - accurev
2145 - GOBject-Builder-gob2
2304 - vimdb.vim
2417 - Merb-and-Datamapper
2502 - fly.vim
2638 - blogger
2651 - fitnesse.vim
2729 - jslint.vim
2838 - HiVim
2871 - FindMate
2885 - gsession.vim
3147 - wmgraphviz

They have been regenerated with everything where it should be. Delete and re-clone!

Small Fixes for options.vim and Emacs Swap Files

2010-11-06T00:00:00-07:00

Some scripts include a file whose name ends in _options.vim that contains example code that should be copied into your ~/.vimrc file. The scraper thought that this was plugin in the wrong place and would move it to the plugin directory. Now it just leaves _options.vim files wherever it finds them.

1352 - Modelines-Bundle
1561 - Rainbow-Parenthsis-Bundle
1609 - Ada-Bundle
1714 - Tail-Bundle
1719 - WLS-Mode
1768 - SnipSnap
2029 - 4NT-Bundle

Also, three scripts included Emacs swap files. The scraper now ignores all files that end in "~".

0100 - TagsBase.zip
2838 - HiVim
2935 - superSnipMate

The affected repos have been regenerated. Delete your local copies and re-clone!

Fixing Duplicate Version IDs

2010-11-06T00:00:00-07:00

The scripts on vim.org are full of releases with the same version number as the previous release. The scraper used to drop these release because it thought it had already imported them.

The scraper now works around this by appending @1 (or @2 if @1 has already been taken, and so on) to the duplicate numbers. They are now unique and git is happy to tag them.

Here's an example: jQuery

The 90 affected scripts have been regenerated:

0015 - ccase.vim
0020 - jcommenter.vim
0030 - python.vim
0031 - a.vim
0072 - foo.vim
0076 - uri-ref
0090 - vcscommand.vim
0096 - methods.vim
0100 - TagsBase.zip
0107 - SelectBuf
0163 - ruby-macros.vim
0164 - html-macros.vim
0171 - multvals.vim
0225 - einstimer.vim
0229 - searchInRuntime
0300 - Vimacs
0360 - rcsdiff.vim
0415 - Zenburn
0465 - vimspell
0508 - ZoomWin
0528 - mh
0561 - help.vim
0563 - rcsvers.vim
0588 - JavaBrowser
0724 - trash.vim
0779 - php_console.vim
0830 - TT2-syntax
0923 - timestamp.vim
0987 - DoxygenToolkit.vim
1038 - vimgrep.vim
1053 - browser.vim
1074 - python_calltips
1139 - tree
1149 - GetFDCText.vim
1213 - Vim-JDE
1268 - acsb
1273 - Mathematica-Syntax-File
1318 - snippetsEmu
1446 - MoveLine
1506 - LargeFile
1535 - TagsParser
1609 - Ada-Bundle
1673 - blueprint.vim
1714 - Tail-Bundle
1770 - etk-vim-syntax
1807 - calmar256-lightdark.vim
1856 - Jinja
2028 - MarkLines
2111 - robocom
2156 - leo256
2158 - diffchanges.vim
2208 - Txtfmt-The-Vim-Highlighter
2257 - bpel.vim
2286 - Specky
2301 - Gembase-dml-plugins
2341 - postmail.vim
2407 - MatlabFilesEdition
2413 - Miranda-syntax-highlighting
2416 - jQuery
2427 - code_complete-new-update
2489 - maroloccio2
2491 - maroloccio3
2497 - shellinsidevim.vim
2512 - guepardo.vim
2535 - cca.vim
2618 - ctk.vim
2620 - neocomplcache
2632 - nimrod.vim
2658 - Changing-color-script
2662 - ywvim
2709 - SudoEdit.vim
2771 - Conque-Shell
2792 - perdirvimrc--Autoload-vimrc-files-per-di
2793 - hybridevel
2831 - mimicpak
2843 - FlipLR
2845 - ywchaos
2945 - AutomaticTexPlugin
2954 - hypergit.vim
2960 - bisect
2995 - colorselector
2999 - django-template-textobjects
3005 - simplecommenter
3056 - Translit3
3081 - JavaScript-Indent
3164 - pythondo
3188 - vimcompcrtr
3199 - vim-addon-ocaml
3244 - Timer-routine
3257 - Say-Time

And script 2792 with that long and truncated name? That is its name on vim.org!