作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.卸载现有的驱动

1.编写剧本

$ cat uninstall_nvidia_580.yml 
---
- name: 彻底卸载NVIDIA 580驱动｜优化无阻塞版（移除卡顿cuda-uninstaller）
  hosts: all
  become: true
  gather_facts: true
  vars:
    cuda_install_path: "/usr/local/cuda-13.0"
    fabric_pkg_name: "nvidia-fabricmanager=580.126.16-1"
  tasks:
    - name: 停止并禁用nvidia-fabricmanager
      service:
        name: nvidia-fabricmanager
        state: stopped
        enabled: false
      ignore_errors: true

    - name: 屏蔽nvidia systemd单元
      shell: systemctl mask nvidia-fabricmanager nvidia-drm
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 彻底purge fabricmanager deb包
      apt:
        name: "{{ fabric_pkg_name }}"
        state: absent
        purge: true
        autoremove: true
      ignore_errors: true

    - name: dpkg兜底清理fabric残留
      shell: timeout 30 dpkg --purge nvidia-fabricmanager
      args:
        executable: /bin/bash
      ignore_errors: true

    # 杀死所有占用GPU设备的进程，释放文件句柄
    - name: 强制kill占用/dev/nvidia*的进程
      shell: |
        timeout 20 fuser /dev/nvidia* 2>/dev/null | awk '{for(i=2;i<=NF;i++) print $i}' | sort -u | xargs -r kill -9
      args:
        executable: /bin/bash
      ignore_errors: true

    # ========== 优化关键：废弃交互卡顿的cuda-uninstaller，直接暴力删除CUDA目录 ==========
    # 注释掉易卡死的cuda-uninstaller整套逻辑
    # - name: 检查cuda-uninstaller是否存在
    #   stat:
    #     path: "{{ cuda_install_path }}/bin/cuda-uninstaller"
    #   register: cuda_uninstall_bin
    #
    # - name: 执行CUDA官方卸载工具（极易IO阻塞，已禁用）
    #   shell: timeout 120 echo y | {{ cuda_install_path }}/bin/cuda-uninstaller
    #   args:
    #     executable: /bin/bash
    #   ignore_errors: true
    #   when: cuda_uninstall_bin.stat.exists

    # 先删除CUDA全部目录，再卸载显卡驱动，避免大量文件占用阻塞卸载脚本
    - name: 批量删除CUDA与NVIDIA系统目录
      file:
        path: "{{ item }}"
        state: absent
      loop:
        - "{{ cuda_install_path }}"
        - /usr/local/cuda
        - /var/lib/nvidia
        - /etc/nvidia
        - /usr/lib/nvidia
        - /usr/lib/x86_64-linux-gnu/nvidia
        - /usr/bin/nvidia-*
        - /etc/modprobe.d/nvidia*.conf
      ignore_errors: true

    - name: 执行显卡驱动卸载脚本 nvidia-uninstall
      shell: timeout 120 echo y | nvidia-uninstall
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 清理所有DKMS nvidia内核模块
      shell: |
        timeout 60 dkms status | grep "^nvidia/" | awk -F'[, ]' '{print $1,$2}' | sort -u | while read mod ver; do
          [ -n "$mod" ] && [ -n "$ver" ] && dkms remove -m "$mod" -v "$ver" --all || true
        done
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 删除DKMS源码目录
      file:
        path: /usr/src/nvidia-*
        state: absent
      ignore_errors: true

    - name: 清理残留nvidia动态链接库
      shell: timeout 30 rm -f /usr/lib/x86_64-linux-gnu/libnvidia*.so*
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 刷新系统ldconfig动态链接缓存
      shell: ldconfig
      args:
        executable: /bin/bash

    - name: 清理/etc/profile CUDA环境变量残留
      shell: timeout 20 sed -i '/cuda\|CUDA_PATH\|LD_LIBRARY_PATH.*nvidia/d' /etc/profile
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 重建initramfs镜像剔除nvidia驱动
      shell: timeout 120 update-initramfs -u
      args:
        executable: /bin/bash

    - name: 更新GRUB引导配置
      shell: timeout 60 update-grub
      args:
        executable: /bin/bash

    - name: 执行主机重启，彻底释放内存中nvidia内核模块
      reboot:
        msg: NVIDIA 580驱动完整清理完成，即将重启主机
      ignore_errors: true

2.执行剧本

$ ansible-playbook -i myhosts uninstall_nvidia_580.yml  


温馨提示:
  在使用该剧本前，建议使用ssh包软件包提前scp到目标主机的/root目录下，不要让ansible拷贝文件，因为驱动单个文件4GB+，直接scp速度会更快，我这里的剧本只是让其做个校验。

  如果不提前拷贝文件到目标主机，速度太慢了。

二.一键升级GPU驱动

1.编写剧本

$ cat install_nvidia_590.yml  
---
- name: 批量安装NVIDIA 590.48.01驱动 + CUDA13.1 + FabricManager（增量+已装自动跳过）
  hosts: all
  become: true
  gather_facts: true
  vars:
    local_workdir: "/yinzhengjie/gpu"
    remote_root: "/root"
    cuda_run_name: "cuda_13.1.1_590.48.01_linux.run"
    fabric_deb_name: "nvidia-fabricmanager_590.48.01-2ubuntu1_amd64.deb"
    cuda_install_path: "/usr/local/cuda-13.1"
    target_driver_ver: "590.48.01"
  tasks:
    # 前置检测当前NVIDIA驱动版本
    - name: 检测当前nvidia驱动版本
      shell: nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null
      args:
        executable: /bin/bash
      register: current_driver_ver
      ignore_errors: true

    - name: 设置标记：是否已安装目标590驱动
      set_fact:
        driver_already_installed: "{{ current_driver_ver.stdout | trim == target_driver_ver }}"

    # 上传CUDA run安装包（删除不兼容的validate_checksum参数）
    - name: 上传CUDA run安装包（文件一致自动跳过）
      copy:
        src: "{{ local_workdir }}/{{ cuda_run_name }}"
        dest: "{{ remote_root }}/{{ cuda_run_name }}"
        mode: '0755'
        force: true

    # 上传FabricManager deb包（删除不兼容的validate_checksum参数）
    - name: 上传FabricManager deb包（文件一致自动跳过）
      copy:
        src: "{{ local_workdir }}/{{ fabric_deb_name }}"
        dest: "{{ remote_root }}/{{ fabric_deb_name }}"
        mode: '0644'
        force: true

    # 仅驱动未安装时，执行CUDA run静默安装
    - name: 静默安装cuda run包（driver + toolkit）
      shell: |
        cd {{ remote_root }}
        ./{{ cuda_run_name }} --silent --driver --toolkit --no-man-page
      args:
        executable: /bin/bash
      register: cuda_install_log
      ignore_errors: false
      when: not driver_already_installed

    - name: 打印CUDA安装日志（便于排错）
      debug:
        msg: "{{ cuda_install_log.stdout_lines }}"
      when: not driver_already_installed

    # 仅驱动未安装时，安装FabricManager
    - name: 安装FabricManager 590.48.01
      shell: dpkg -i {{ remote_root }}/{{ fabric_deb_name }}
      args:
        executable: /bin/bash
      register: fabric_install
      ignore_errors: true
      when: not driver_already_installed

    # 仅驱动未安装时，修复deb依赖缺失
    - name: 修复deb依赖缺失
      shell: apt-get -yf install
      args:
        executable: /bin/bash
      ignore_errors: false
      when: not driver_already_installed

    # 无论是否已装驱动，都更新全局CUDA环境变量
    - name: 写入CUDA环境变量到/etc/profile
      blockinfile:
        path: /etc/profile
        marker: "# {MARKER} CUDA 13.1 590 Driver Env"
        block: |
          export CUDA_HOME={{ cuda_install_path }}
          export PATH=$CUDA_HOME/bin:$PATH
          export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH

    - name: 刷新动态链接库缓存
      shell: ldconfig
      args:
        executable: /bin/bash

    # 加载IB依赖内核模块 ib_umad 解决fabricmanager启动失败
    - name: 加载ib_umad内核模块，FabricManager依赖
      shell: modprobe ib_umad
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 配置开机自动加载ib_umad
      copy:
        content: "ib_umad"
        dest: /etc/modules-load.d/nvidia-fabric-ib.conf
        mode: '0644'

    # 取消mask、启动并开机自启fabricmanager（必执行，修复之前mask问题）
    - name: 取消mask并启动nvidia-fabricmanager，开机自启
      shell: |
        systemctl unmask nvidia-fabricmanager
        systemctl daemon-reload
        systemctl start nvidia-fabricmanager
        systemctl enable nvidia-fabricmanager
      args:
        executable: /bin/bash
      ignore_errors: true

    - name: 验证nvidia-smi输出
      shell: nvidia-smi
      args:
        executable: /bin/bash
      register: nvidia_smi_res

    - name: 打印nvidia-smi信息
      debug:
        msg: "{{ nvidia_smi_res.stdout_lines }}"

    # 全路径校验nvcc，修复YAML引号语法
    - name: 验证CUDA Toolkit nvcc编译器（全路径）
      shell: "{{ cuda_install_path }}/bin/nvcc -V"
      args:
        executable: /bin/bash
      register: nvcc_res

    - name: 打印nvcc版本
      debug:
        msg: "{{ nvcc_res.stdout_lines }}"

2.测试验证

$ ansible-playbook -i myhosts install_nvidia_590.yml